Convert Video to VR

I thought of a neat piece of software I would like to see:
This application would extract the background from a video stream that included pans and tilts in the shot.
Say you have a shot of someone running across a field. The software would process the shot into a VR background image. Using pattern recognition, each frame would add to the background image as the camera panned across a new region. Only the background would survive this process. If there was an object being tracked in the shot, then this object would not be part of the background "plate".
An extension to this process would be the ability to derive a new view of the object moving across a static background. It would be possible to view the object from a select range of angles. This part would be a little harder if the original object was near enough to the camera so that the camera saw multiple sides of the object. But this could all be sorted out.
This idea came out of my thoughts of how my video and image collection could be used to create a virtual world for myself. Sometime in the future, all of my media will be digitized and used to create a space for me to explore my life's history. People from my past will be recreated using AI algorithms and every bit of video of that person. The AI algorithm would be able to simulate the person, so that person would live on in my memory.
Eventually this data will be integrated with my own memories. Even more, my memories will be mined by the software to help me recollect things that are only dim memories for me now.
Then that first kiss, the bee sting I got when I was three - and so on - would be as real as new.