2D Video to VR Environment and Characters

For the last few months, I've been thinking a lot about what I would ultimately do with the hours of video I have taken over the course of my children's lives so far. I can't imagine actually watching all of that video, or even trying to edit it down. I don't really want to experience it in a linear fashion. After all, I was there. And while I may have experienced it through the viewfinder of a camera, still - I was there.

I've concluded that what would be useful would be to have a computer watch the video for me!
The computer would interpret the video, identifying the time and place, capturing the environment, and digitizing the people, creating photo-realistic 3D avatars of them.
In the long run, these videos will eventually be incorporated into my own (augmented) memory. The people's behaviors would be catalogued and pattern-recognized to the point that realistic simulations of the people - at various ages - could be made. I could have conversations and interactions with those who were no longer with me.

I can see that, ultimately, the computer's AI will be sufficient to really interpret the videos as well as I could if I were watching them. It would generate new memories, very similarly to what would happen to me - my memory "refreshed" - if I were to watch them.

Of course the computers of today aren't quite there yet. They are only now able to recognize the environment well enough to drive at about 14 miles an hour.

I would really like to see this technology developed, and I would like to know as much about it as I can. I have started (in my mid-life now) reading up on projective geometry, C++ programming and so on so that I can do some hobbyist-level playing around with this technology. I look forward to the day when the tools start to exist that would enable me to begin tackling these piles of videos I have here at home.

Thanks to a fellow named Augusto Roman (thanks for the link!), I now have a tool that will get me started. It's called Voodoo. It's camera tracking software that will "watch" a video clip and create both a point cloud of what is in the video, and a camera path. Both of these things can be exported to Blender, a free (and quite powerful) 3D animation package. I am now planning to try this with a few video clips. Of course, the package assumes a static scene with just the camera moving (if I understand it correctly), so I know it (by itself) isn't going to get me to the goal of separating out the moving objects from the environment. But it is a start.

Voodoo Camera Tracker is at: http://www.digilab.uni-hannover.de/docs/manual.html