Okay, here we go.
Neural Volume Rendering for Dynamic Scenes
NeRF has shown incredible view synthesis results, but it requires multi-view captures for STATIC scenes.
How can we achieve view synthesis for DYNAMIC scenes from a single video? Here is what I learned from several recent efforts.
Okay, here we go.
NeRF represents the scene as a 5D continuous volumetric scene function that maps the spatial position and viewing direction to color and density. It then projects the colors/densities to form an image with volume rendering.
Volumetric + Implicit -> Awesome!
Building on NeRF, one can extend it for handling dynamic scenes with two types of approaches.
A) 4D (or 6D with views) function.
One direct approach is to include TIME as an additional input to learn a DYNAMIC radiance field.
e.g., Video-NeRF, NSFF, NeRFlow
Inspired by non-rigid reconstruction methods, this type of approach learns a radiance field in a canonical frame (template) and predicts deformation for each frame to account for dynamics over time.
e.g., Nerfie, NR-NeRF, D-NeRF
All the methods use an MLP to encode the deformation field. But, how do they differ?
A) INPUT: How to encode the additional time dimension as input?
B) OUTPUT: How to parametrize the deformation field?
One can choose to use EXPLICIT conditioning by treating the frame index t as input.
Alternatively, one can use a learnable LATENT vector for each frame.
We can either use the MLP to predict
- dense 3D translation vectors (aka scene flow) or
- dense rigid motion field
More from Tech
I think about this a lot, both in IT and civil infrastructure. It looks so trivial to “fix” from the outside. In fact, it is incredibly draining to do the entirely crushing work of real policy changes internally. It’s harder than drafting a blank page of how the world should be.
I’m at a sort of career crisis point. In my job before, three people could contain the entire complexity of a nation-wide company’s IT infrastructure in their head.
Once you move above that mark, it becomes exponentially, far and away beyond anything I dreamed, more difficult.
And I look at candidates and know-everything’s who think it’s all so easy. Or, people who think we could burn it down with no losses and start over.
God I wish I lived in that world of triviality. In moments, I find myself regretting leaving that place of self-directed autonomy.
For ten years I knew I could build something and see results that same day. Now I’m adjusting to building something in my mind in one day, and it taking a year to do the due-diligence and edge cases and documentation and familiarization and roll-out.
That’s the hard work. It’s not technical. It’s not becoming a rockstar to peers.
These people look at me and just see another self-important idiot in Security who thinks they understand the system others live. Who thinks “bad” designs were made for no reason.
Who wasn’t there.
The tragedy of revolutionaries is they design a utopia by a river but discover the impure city they razed was on stilts for a reason.
— SwiftOnSecurity (@SwiftOnSecurity) June 19, 2016
I’m at a sort of career crisis point. In my job before, three people could contain the entire complexity of a nation-wide company’s IT infrastructure in their head.
Once you move above that mark, it becomes exponentially, far and away beyond anything I dreamed, more difficult.
And I look at candidates and know-everything’s who think it’s all so easy. Or, people who think we could burn it down with no losses and start over.
God I wish I lived in that world of triviality. In moments, I find myself regretting leaving that place of self-directed autonomy.
For ten years I knew I could build something and see results that same day. Now I’m adjusting to building something in my mind in one day, and it taking a year to do the due-diligence and edge cases and documentation and familiarization and roll-out.
That’s the hard work. It’s not technical. It’s not becoming a rockstar to peers.
These people look at me and just see another self-important idiot in Security who thinks they understand the system others live. Who thinks “bad” designs were made for no reason.
Who wasn’t there.
The entire discussion around Facebook’s disclosures of what happened in 2016 is very frustrating. No exec stopped any investigations, but there were a lot of heated discussions about what to publish and when.
In the spring and summer of 2016, as reported by the Times, activity we traced to GRU was reported to the FBI. This was the standard model of interaction companies used for nation-state attacks against likely US targeted.
In the Spring of 2017, after a deep dive into the Fake News phenomena, the security team wanted to publish an update that covered what we had learned. At this point, we didn’t have any advertising content or the big IRA cluster, but we did know about the GRU model.
This report when through dozens of edits as different equities were represented. I did not have any meetings with Sheryl on the paper, but I can’t speak to whether she was in the loop with my higher-ups.
In the end, the difficult question of attribution was settled by us pointing to the DNI report instead of saying Russia or GRU directly. In my pre-briefs with members of Congress, I made it clear that we believed this action was GRU.
The story doesn\u2019t say you were told not to... it says you did so without approval and they tried to obfuscate what you found. Is that true?
— Sarah Frier (@sarahfrier) November 15, 2018
In the spring and summer of 2016, as reported by the Times, activity we traced to GRU was reported to the FBI. This was the standard model of interaction companies used for nation-state attacks against likely US targeted.
In the Spring of 2017, after a deep dive into the Fake News phenomena, the security team wanted to publish an update that covered what we had learned. At this point, we didn’t have any advertising content or the big IRA cluster, but we did know about the GRU model.
This report when through dozens of edits as different equities were represented. I did not have any meetings with Sheryl on the paper, but I can’t speak to whether she was in the loop with my higher-ups.
In the end, the difficult question of attribution was settled by us pointing to the DNI report instead of saying Russia or GRU directly. In my pre-briefs with members of Congress, I made it clear that we believed this action was GRU.
