2D Video to VR Environment and Characters

For the last few months, I've been thinking a lot about what I would ultimately do with the hours of video I have taken over the course of my children's lives so far. I can't imagine actually watching all of that video, or even trying to edit it down. I don't really want to experience it in a linear fashion. After all, I was there. And while I may have experienced it through the viewfinder of a camera, still - I was there.

I've concluded that what would be useful would be to have a computer watch the video for me!
The computer would interpret the video, identifying the time and place, capturing the environment, and digitizing the people, creating photo-realistic 3D avatars of them.
In the long run, these videos will eventually be incorporated into my own (augmented) memory. The people's behaviors would be catalogued and pattern-recognized to the point that realistic simulations of the people - at various ages - could be made. I could have conversations and interactions with those who were no longer with me.

I can see that, ultimately, the computer's AI will be sufficient to really interpret the videos as well as I could if I were watching them. It would generate new memories, very similarly to what would happen to me - my memory "refreshed" - if I were to watch them.

Of course the computers of today aren't quite there yet. They are only now able to recognize the environment well enough to drive at about 14 miles an hour.

I would really like to see this technology developed, and I would like to know as much about it as I can. I have started (in my mid-life now) reading up on projective geometry, C++ programming and so on so that I can do some hobbyist-level playing around with this technology. I look forward to the day when the tools start to exist that would enable me to begin tackling these piles of videos I have here at home.

Thanks to a fellow named Augusto Roman (thanks for the link!), I now have a tool that will get me started. It's called Voodoo. It's camera tracking software that will "watch" a video clip and create both a point cloud of what is in the video, and a camera path. Both of these things can be exported to Blender, a free (and quite powerful) 3D animation package. I am now planning to try this with a few video clips. Of course, the package assumes a static scene with just the camera moving (if I understand it correctly), so I know it (by itself) isn't going to get me to the goal of separating out the moving objects from the environment. But it is a start.

Voodoo Camera Tracker is at: http://www.digilab.uni-hannover.de/docs/manual.html

Things to do

There is a multitude of things I don't understand, that I would like to. I want to move along in converting my photo and video archives into a VR memory-assist library or museum. Initially, this VR space will occupy a computer memory. Eventually, I hope that it will be incorporated into my own augmented brain. This seems to be the subject this blog is taking, somewhat without my having planned it that way.
I have begun to realize that, in a way, EVERYTHING is software. That means that without the personal ability to create and modify software, I am without a very necessary skill set. I have begun to study the C++ programming language. I downloaded the free text of "Thinking in C++" and I am studying that every day. In addition, I have discovered many articles about how various groups have implemented the conversion of video streams into panoramic mosaics and even full 3D environments. Imagine being able to video record a scene, and then feed that video into an application that converts it into a 3D space. That's what is coming. If you are interested, you should look up the Photosynth website. This is a Microsoft Research product that correlates any number of photographs of an area into a 3D space, and retains the ability to navigate to each image. The demo video is really cool.

The Internet as Personal Memory

Well, the Internet is going to become my audio/video/photographic memory. My flickr account lets me store my photos online. Soon I will start uploading videos to Google Video (or somewhere) so the whole world can watch my home movies....

You know, there's a touch of this in Bruce Sterling's short story, "Maneki Neko". It's a great story, and can be found in his anthology "A Good, Old-Fashioned Future".

I'm continuing my move to the use of all Open software. I have one Linux box up and running. It was rescued from a relative, had XP home on it, and was almost completely trashed by viruses. Since it came to me for free, I didn't feel like I was risking anything by formatting the disk and starting over.