"explorable" and "immersive" is definitely a bold choice of words when you can't really get below the level of the buildings before the gaussian splatting is very obvious. Sure, it's impressive that you can get that detailed from a few satellite images, but I think that might be overselling it a bit
> you can't really get below the level of the buildings before the gaussian splatting is very obvious.
True. If you bring the viewpoint down to near street level and look horizontally, it's worse than traditional photogrammetry methods.
I've been looking for algorithms like this for representing distant regions in virtual worlds. Open Drone Map can do a good job, sometimes, but it really needs a cleanup pass.
That's a matter of data, and this looks promising to me. If instead of satellite images they'd feed it drone shots they could probably get down to a level of detail that becomes actually immersive and would be way beyond the digital twins we currently have.
We probably won't. GS is a reconstructive method, so when data is unavailable, you can only perform poor interpolation. You would need additional generative, not reconstructive, models. However, this would open the door to unfaithful augmentation again.
You’d be surprised what GIS - or at least GIS - adjacent customers want. If you think about any cute-but-useless map detail that comes to your mind there is likely a paying customer for it.
If you want to dig more, I suspect Microsoft got a lot (if not all) of it from blackshark.ai . They're one of the companies whose logo shows up during game start.
I knew their name because, when I worked for an Airbus subsidiary, we talked with them about a solution to generate 3D environments for any/every airport.
They had some cool stuff but also some wonky stuff at the time (like highway overpasses actually being rendered as walls across the highway).
blackshark.ai was used for generating building models with textured facades from footprints. Their thing for generating airports was actually not used in MSFS. The photogrammetry from satellite photos in MSFS 2024 was done by Maxar Technologies.
Maybe dumb question but how do I just take a sat image and create the scene? The scripts in the repo are all about training which I assume requires you to have the 3d data too.
These sort of projects always look cool but I think the real "wow factor" would be a file upload where you can see the result on your image. I assume there are reasons why this isn't done.
Do you mean converting your image into a 3D scene?
This is where we were heading with our 3D volumetric video company https://ayvri.com
We were working on blending 3D satellite imagery with your ground view (or low flying in the case of paragliders) photos and videos to create a 3D scene.
Our technology was acquired prior to us being able to fully realize the vision (and we moved on to another project).
This is so cool. I used to work on urban heat island analysis and now work in natural catastrophe modelling, and in both cases knowing the average heights/volumes of buildings is a very handy thing to have but is surprisingly difficult information to retrieve. Even a coarse estimate available at annual resolution has some really awesome use cases, very excited to see this.
It looks good! I imagine a reasonable next step might be to do something about the cars, which are omnipresent in urban scenes but seem like they've been left a blurry mess in the examples.
MSFS 2024 already does photogrammetry from satellite photos. However, it builds triangle geometry much like is done from aerial photography, because gaussian splats are not suitable for games; you can't build collision geometry from a gaussian splat for example.
In fact you wouldn't even need to be limited to earth. Why not throw in Google Moon and steal a moon buggy while shooting scientific rovers and doing cool flips out of craters?
There are already 3D globes of the moon, for example with cesiumjs: https://sandcastle.cesium.com/?id=moon you can even import it to unity and other engines
Nice, but when you look up close things like this and Google Earth look like a post-apocalyptic scene :)
It would be amazing if they could also take user-generated photos and videos at ground level and accurate mapping data (that has building outlines) and clean that up to something presentable.
I mean, what they do here is what google and apple are already doing for years. It's time for the next step.
I suspect hybrid solutions will remove the limitations of GS, with (eventually...) some smooth hand off. Do clean-enough GS like this; then hand the output to other systems which covert into forms more useful for your application and which adopt e.g. textures from localized photos etc.
"explorable" and "immersive" is definitely a bold choice of words when you can't really get below the level of the buildings before the gaussian splatting is very obvious. Sure, it's impressive that you can get that detailed from a few satellite images, but I think that might be overselling it a bit
> you can't really get below the level of the buildings before the gaussian splatting is very obvious.
True. If you bring the viewpoint down to near street level and look horizontally, it's worse than traditional photogrammetry methods.
I've been looking for algorithms like this for representing distant regions in virtual worlds. Open Drone Map can do a good job, sometimes, but it really needs a cleanup pass.
That's a matter of data, and this looks promising to me. If instead of satellite images they'd feed it drone shots they could probably get down to a level of detail that becomes actually immersive and would be way beyond the digital twins we currently have.
We're early days. Models will soon interpolate all of that. Eventually in real time.
I wouldn't knock the research. The results look impressive to me.
We probably won't. GS is a reconstructive method, so when data is unavailable, you can only perform poor interpolation. You would need additional generative, not reconstructive, models. However, this would open the door to unfaithful augmentation again.
Different applications.
GIS won't want generative hallucinations.
Consumer mapping apps, social applications, and games (eg. flight sims) will want the maps to look as good as possible.
GIS don't want half exploded buildings either. Nor would they care about photographic textures on the 3D models.
You’d be surprised what GIS - or at least GIS - adjacent customers want. If you think about any cute-but-useless map detail that comes to your mind there is likely a paying customer for it.
While the methodology wasn't published, Microsoft did something similar for Flight Simulator.
https://www.smithsonianmag.com/air-space-magazine/flight-box...
If you want to dig more, I suspect Microsoft got a lot (if not all) of it from blackshark.ai . They're one of the companies whose logo shows up during game start.
I knew their name because, when I worked for an Airbus subsidiary, we talked with them about a solution to generate 3D environments for any/every airport.
They had some cool stuff but also some wonky stuff at the time (like highway overpasses actually being rendered as walls across the highway).
blackshark.ai was used for generating building models with textured facades from footprints. Their thing for generating airports was actually not used in MSFS. The photogrammetry from satellite photos in MSFS 2024 was done by Maxar Technologies.
Maybe dumb question but how do I just take a sat image and create the scene? The scripts in the repo are all about training which I assume requires you to have the 3d data too.
These sort of projects always look cool but I think the real "wow factor" would be a file upload where you can see the result on your image. I assume there are reasons why this isn't done.
Do you mean converting your image into a 3D scene?
This is where we were heading with our 3D volumetric video company https://ayvri.com
We were working on blending 3D satellite imagery with your ground view (or low flying in the case of paragliders) photos and videos to create a 3D scene.
Our technology was acquired prior to us being able to fully realize the vision (and we moved on to another project).
This is so cool. I used to work on urban heat island analysis and now work in natural catastrophe modelling, and in both cases knowing the average heights/volumes of buildings is a very handy thing to have but is surprisingly difficult information to retrieve. Even a coarse estimate available at annual resolution has some really awesome use cases, very excited to see this.
It looks good! I imagine a reasonable next step might be to do something about the cars, which are omnipresent in urban scenes but seem like they've been left a blurry mess in the examples.
Is there any reason that this couldn't integrate Street View data?
This would be the next step for flight simulators, which while remarkable still require handmade assets for accurate details.
MSFS 2024 already does photogrammetry from satellite photos. However, it builds triangle geometry much like is done from aerial photography, because gaussian splats are not suitable for games; you can't build collision geometry from a gaussian splat for example.
afaik msfs uses partially automatically generated 3d assets (from Bing Maps?) from aerial/satellite imagery.
Very cool; interesting how it turns all the trees into puffballs though. Some artifact of the pre-trained depth estimation or diffusion model maybe?
This could be specially good for a world 3d model for flightgear.
Once Flightgear could use Google Earth assets.
Now the GTA: Anywhere please...
Ya, can't wait to play GTA: Nova Zemyla.
In fact you wouldn't even need to be limited to earth. Why not throw in Google Moon and steal a moon buggy while shooting scientific rovers and doing cool flips out of craters?
There are already 3D globes of the moon, for example with cesiumjs: https://sandcastle.cesium.com/?id=moon you can even import it to unity and other engines
Nice, but when you look up close things like this and Google Earth look like a post-apocalyptic scene :)
It would be amazing if they could also take user-generated photos and videos at ground level and accurate mapping data (that has building outlines) and clean that up to something presentable.
I mean, what they do here is what google and apple are already doing for years. It's time for the next step.
> I mean, what they do here is what google and apple are already doing for years
This is gaussian splatting. I'm pretty confident that google/apple have not done that.
Oh I didn't realise it was a different technique but the result is similarly bad when zoomed in :(
Re: utility in games,
I suspect hybrid solutions will remove the limitations of GS, with (eventually...) some smooth hand off. Do clean-enough GS like this; then hand the output to other systems which covert into forms more useful for your application and which adopt e.g. textures from localized photos etc.
It's just a bit of engineering and compute...