The software of my dreams

Hello, I created an account just to let you know that there is a piece of software called RenderDoc that is commonly used for graphics debugging. It lets you run any program in it and capture graphics data, then view the geometry, textures, shaders, framebuffers, etc. from that capture file.

Not exactly what you were talking about, but using photogrammetry like in this video to port maps is something others have done.

Photogrammetry is usually used by devs to get IRL locations into an engine but it should be able to work with games.

This is the video that finally pulled me into creating an account here, so well done. Love the idea, and it really gets the brain juices going.

 

To make sure I'm understanding the idea right: Ross is proposing brute-forcing the ripping of 3d model and texture data (mostly for environments) by using an AI to analyze footage of the exploration of a game space and then attempt to rebuild it in an easily usable engine format (say unity or unreal). The advantage of this is obviously that it's universal, but it's also a hell of an ask (and obviously Ross is aware of this).

 

However, in the spirit of going for the low-hanging fruit first, wouldn't it make more sense to just make a tool that collects, automates and standardizes ripping tools? If you have the right tools for a given engine, it's very do-able to just rip map data, models, animation, lighting etc, convert it over to a standardized format and re-assemble the pieces into a working space again (in an automated, push-button way no less), no AI magic required.  Given, this would require individual attention and problem-solving on a per-engine (and maybe even per-game) basis, but fans are already doing that on a regular basis for popular games, and it would be worlds easier than trying to wrangle an AI or work with some kind of simulated photogrammetry.

 

The website in the video showing off reconstructed Nintendo worlds shows how viable this approach could be. The real opportunity as I see it is standardizing and automating what's already out there, and what has yet to be created in terms of ripping tools. Right now if you wanted pull the worlds from say, the new Spyro remake, that's totally doable, fans have created the tools to largely re-construct and pull it all back into an Unreal project. But it's also a nightmarish hodgepodge of "use this and then this and then follow these obscure steps and then-" etc.  If someone took all those tools, wove them together with some scripting and slapped a nice UI on top so it was just "Rip game map" "Play game map" or whatever, there's the dream software.

 

This still takes real time and effort obviously, and it would have to happen in pockets: only games with a community devoted enough to build ripping tools and a plugin for this "universal" program would be accessible. But again, low hanging fruit. To steal something Ross said in the video: something is better than nothing. Ripping and reconstructing like this would be infinitely easier and more practically do-able than trying to mess with AI.

 

Just some thoughts though. Like I said, that's why these videos are great, they get you thinking. Here's another thought: if this software was created, it could potentially ride the "just barely legal" wave in the same way emulators do. You can't share the maps, but that doesn't matter so much if anyone can create the maps themselves, at home, as long as they have both the software and the game.

Great video, Ross! I don't think about this kind of thing often, but you know how to sell an imaginary concept. Virtual world exploring does sound lovely, especially if there's surround sound ambience and such.

 

I hope this dream software doesn't distract the programmers in the audience away from developing a better GUI though; that would be the software that would further the human race.

@Ross Scott

Jasper, the creator of that noclip site you mentioned near the end, made a comment on your video regarding the feasibility of your idea. It might be best if you responded directly to them in the YouTube comments section, but here's a copy of said comment for prosperity's sake:

 

Quote

tl;dr: You want Garry's Mod.

 

Hi there, I'm the creator of noclip.website, so I'd say I know quite a bit about the internals of different games. Let's get this straight out of the way: from a technical perspective, this idea is nearly impossible. Compelling, of course, but the magic behind games is a lot more tricky, nuanced, and special-cased than I think you imagine.

 

To use your "take Train Simulator and add zombies" example: I haven't played Train Simulator, but from the footage I've seen, the train's on rails. Do you think that game even bothered with collision data, or AI pathing data for the zombies to run on? What would the zombies even do? All the zombies I can think of in games are encountered in close hallways to ambush the player; so they move slowly, their AI is designed to trap you, and when they get close, they stand in one spot and hit heavy. Spawning a few in Train Simulator, well, someone has to animate and program them being shoved out of the way when hit by a train. To spawn in a zombie from Half-Life 2 and just expect it to "just work" is setting up a scary expectation for games and for developers. I'd say we're pretty good at our jobs, but not that good.

 

Not to mention that this workload only goes up as we add more elements. If we add, say, Octoroks in, now we have to plan for three "pairs" of interactions: Zombie/Train, Zombie/Octorok, Train/Octorok. With 4 elements, that's 6 pairs. So, in a way, adding "just one more thing" is the hardest part; and most times this happens, developers don't plan on supporting all interactions. In your software, where the player can do literally anything with any game, you are committed to implementing all of that.

 

I think a lot of early game developers start out with "I will make a game where you can do anything" as an idea (I was that way once as well!), but quickly learn that the constraints in the game's design are what make it fun. The elegance of Portal was everything they didn't add, it meant it was possible to keep the entire game in your head.

 

Garry's Mod is probably the closest we'll ever get to your dream software, and it's great! I love Garry's Mod, but it's still very limited in its scope compared to what you want, despite having probably the biggest scope of any of these tools. And it still doesn't come with fun out of the box, you have to add it in! Thankfully, there's a huge community doing that, and so you can play Prop Hunt in Peach's Castle from Super Mario 64 if you want to.

 

Don't get me wrong, some of these levels and worlds feel so real, feel so close, and I totally understand how compelling it is to feel transported to somewhere, and want to stay longer. That's why I originally built noclip.website, after all! So I understand your passion here, I just want you to know what you're asking for here, lol.

 

5 minutes ago, SomethingOther said:

@Ross Scott

Jasper, the creator of that noclip site you mentioned near the end, made a comment on your video regarding the feasibility of your idea. It might be best if you responded directly to them in the YouTube comments section, but here's a copy of said comment for prosperity's sake:

Oh God, it's like the response to the GaaS video all over again.

16 hours ago, ScumCoder said:

Oh God, it's like the response to the GaaS video all over again.

This one is on me though, I explained this part poorly.  I left a pinned comment to clear it up.  Basically, he would be right if that's what I was advocating, but I'm not, but i could easily see where people thought that.

 

On 1/9/2021 at 8:42 AM, TangledTwine said:

This is the video that finally pulled me into creating an account here, so well done. Love the idea, and it really gets the brain juices going.

 

To make sure I'm understanding the idea right: Ross is proposing brute-forcing the ripping of 3d model and texture data (mostly for environments) by using an AI to analyze footage of the exploration of a game space and then attempt to rebuild it in an easily usable engine format (say unity or unreal). The advantage of this is obviously that it's universal, but it's also a hell of an ask (and obviously Ross is aware of this).

 

However, in the spirit of going for the low-hanging fruit first, wouldn't it make more sense to just make a tool that collects, automates and standardizes ripping tools? If you have the right tools for a given engine, it's very do-able to just rip map data, models, animation, lighting etc, convert it over to a standardized format and re-assemble the pieces into a working space again (in an automated, push-button way no less), no AI magic required.  Given, this would require individual attention and problem-solving on a per-engine (and maybe even per-game) basis, but fans are already doing that on a regular basis for popular games, and it would be worlds easier than trying to wrangle an AI or work with some kind of simulated photogrammetry.

 

The website in the video showing off reconstructed Nintendo worlds shows how viable this approach could be. The real opportunity as I see it is standardizing and automating what's already out there, and what has yet to be created in terms of ripping tools. Right now if you wanted pull the worlds from say, the new Spyro remake, that's totally doable, fans have created the tools to largely re-construct and pull it all back into an Unreal project. But it's also a nightmarish hodgepodge of "use this and then this and then follow these obscure steps and then-" etc.  If someone took all those tools, wove them together with some scripting and slapped a nice UI on top so it was just "Rip game map" "Play game map" or whatever, there's the dream software.

 

This still takes real time and effort obviously, and it would have to happen in pockets: only games with a community devoted enough to build ripping tools and a plugin for this "universal" program would be accessible. But again, low hanging fruit. To steal something Ross said in the video: something is better than nothing. Ripping and reconstructing like this would be infinitely easier and more practically do-able than trying to mess with AI.

 

Just some thoughts though. Like I said, that's why these videos are great, they get you thinking. Here's another thought: if this software was created, it could potentially ride the "just barely legal" wave in the same way emulators do. You can't share the maps, but that doesn't matter so much if anyone can create the maps themselves, at home, as long as they have both the software and the game.

I generally just assumed this method would be a never-ending hell with standards changing all the time, encrypted files, and only the most popular games getting support.  Even games that use the same engine can have massive compatibility differences.    A ripping-solution from the output would be something that would only get refined over time and not have to keep reinventing the wheel, but I'm happy for whatever gets us more worlds.

5 hours ago, Ross Scott said:

I generally just assumed this method would be a never-ending hell with standards changing all the time, encrypted files, and only the most popular games getting support.  Even games that use the same engine can have massive compatibility differences.    A ripping-solution from the output would be something that would only get refined over time and not have to keep reinventing the wheel, but I'm happy for whatever gets us more worse

 

The lack of support for games without a big community around them is a big downside, yes. If you're the only one who cares about a given game, my version wouldn't really help much.  The other stuff isn't as much of an issue though. Yes, games are a moving target, but if the fans care they're usually pretty good at hitting that target. The core scripting engine extender for Skyrim has to be updated every time Bethesda updates the game's exe, which for a while was VERY often, but the fans managed to keep it updated. Not saying it wouldn't be hard, it definitely would be. But it would still probably be easier than the AI approach. That said, I'd love to be wrong.

 

Also, realized I hadn't answered the call for what MY dream software is, so here goes: an OS that "programs" itself. Monitors the web for news of security and malware issues, parses the text with a high level of understanding, and patches vulnerabilities (complete with changelog). Analyzes unfamiliar hardware and can gradually write its own drivers for said hardware, via research and testing. a-b tests new GUI ideas on the User: User is trying to find a file? Some days the dialog is a little more like this, others a little more like that, collect data on which seems more effective and enjoyed by the user, implement (major sites like google and facebook do this all the time, this would "just" be without a human in the loop).  Fairly out there/sci fi at the moment, but within the horizon of possibility in a decade or so I think.

I'm glad to see Ross is well after traversing a Doom 3 hell portal at the end of the video.

I think for games just using the video input is just the first step - the extra steps of segmenting the data and getting individual models out of it would be too messy/lossy to be easily usable without doing some actual archaeology level work and and cataloging / labeling objects and such. The AI would have no context for this just from videos.

 

The "NES games to 3D" approach where the AI actually looks at and tries to understand the game code/memory a bit would be more likely to get usable / quality data. Applying that to 3D games might not be theoretically impossible, but it's of course not actually feasible because the search space is so much bigger and everything is much more complex.

 

If instead of the video the input would be a screenshot and a dump of the games RAM/VRAM to search. It could try to find the textures/3d data in the memory that correlate with the input image. It'd still require doing photogrammetry on the source image so the AI would know exactly what it's trying to find and what transforms/camera angles are needed. I really just want the raw data (textures/geometry) to see how the original creators made it in the first place. An AI created imitation doesn't give me that.

 

It might also be a less open-ended problem with clear goals to train an AI for - recreate this image in a 3d engine using that memory data.

 

Another and actual possible-to-implement variant of this would be something like an asset flip detector that matches to a catalog of known models and gives you the models from that instead of reverse engineering anything. Like a reverse image search but for 3D models. It's cheating of course and wouldn't give you any of the the interesting and unique stuff. From that idea you could maybe even work towards recreating game worlds using generic assets instead of exact matches - because usually doesn't really matter if it's rock_03b or rock_7d.

 

I haven't played it, but there's a game* that feels adjacent to this whole thing of visiting other worlds called Jazztronauts where you steal props from random Source engine maps. The concept of stealing anything from any game speaks to me.

 

* it's a mod for Garry's mod

The first thing I did when I bought a VR headset was to download the Star Trek Enterprise tour that was shut down a few years ago for copywright. 

What I mean is, I get you, Ross The Shaper

Yo ross, this video really got to me and I simply have to comment.

What i am truly interested is, are you REALLY interested in helping this kind of project going?

I write as a work with photogrametry, I used to be a researcher that worked in plant phenotyping and used technology like the ones that several people already commented in this thread. (WAS as bad economy left me jobless, but moved to create a business for 3d reconstructions from plants for farmers using cellphone and drone images)

In reality, just free software from more than 7 years ago can be used to extract 3d data from worlds and create files that can be read by any engine.

What occurs is that in practice, you dont need video, you would need thousands of images taken in specific automated manners, and then you could clean results manually, or use trainers for machine learning. Sky is the limit here.

 

What i think is a collaborative effort to extract data, like is normally done in cultural heritage, from which things like rome have been preserved in 3d thanks to images from tourists online.

We could call it, gaming heritage.

However for a project of this magnitude, several points must be met:

First, scratch the idea of software for extracting data. Also, the rest of the dream engine is way too far fetched.

Focus on getting said data in the first place as a collaborative effort.

Call on people to take images of games, upload them into a massive file-sharing service, and have a group of people convert those into 3d meshes. Share the results. I'm skipping TONS of technical details

As it is transformative work is shouldn't be illegal (im no lawyer, however)

Those results should be easily accessible for plenty of viewers, so people can explore said worlds.

You said that you cannot do much because you are out of your league, this is false. You can make the call to make people try to collaborate on this, kickstart it somehow. I would really love to even explain the basics behind some of what structure-from-motion is, so let me know.

Sigh. As someone who wrote the kind of software that takes a video and spews out 3D models, i can say that this is hellishly hard to do, and what you tend to see in demos often are carefully picked best case scenarios.

 

What i would have loved to have and was working towards is VR photography - take several 360 pictures or a video of a place, and receive a 3D model of that place to look around and/or walk around in VR. But without all the busywork and preparations, so i would actually take such pictures on the go instead of going on dedicated expeditions with several pieces of rare equipment and a tangle of wires, followed by being too lazy to actually do the processing and cleanup.

 

The technique would work well in a game world, but one thing i didn't notice you or anyone mention is that it would take A LOT of time. A game world can be vast, and recreating it via video means you have to cover all the ground, at a distance close enough for the finest details to load in. For a game that lets you drive all over a continent to scale this would be an untractable job.

 

On reverse rendering it's basically what it says on the tin - you have lights that emit darkness. You do the geometry calculations, but instead of writing the texture to the image you read from image to texture. That's how i did texture extraction in my thingy. So removing shadows shouldn't be too hard if you can identify light sources and/or determine shadow mapping parameters. Removing more modern ambient lighting and raytraced lighting might be much harder, however.

 

Also, i like the name. Dream software. Dreams are worlds you wander around. Lucid dreams are ones you where you do it deliberately. Nice fit.

 

56 minutes ago, Artlav said:

Sigh. As someone who wrote the kind of software that takes a video and spews out 3D models, i can say that this is hellishly hard to do, and what you tend to see in demos often are carefully picked best case scenarios.

Could video game capture be one of those best case scenarios with a stable camera a vastly simplified light model (pre raytracing at least) and repeated materials/textuers. Going back to my previous post where I mentioned reading the games memory - reading the exact camera position/direction/fov from the memory would also be more accurate than gyroscope data and could help any algorithms.

 

I get the feel that in the real world AI is mainly used to make sense of messy data and is basically statistics if you dig deep enough -- but video/other data from games could be way less messy if you think about what you're feeding it - disabling bloom/motion blur/other fancy camera effects could make the (still impossible) task much easier if you think of it as a specific problem incompatible with real world video-to-3d.

 

 

On 1/12/2021 at 8:08 AM, Ovimeister said:

Yo ross, this video really got to me and I simply have to comment.

What i am truly interested is, are you REALLY interested in helping this kind of project going?

I write as a work with photogrametry, I used to be a researcher that worked in plant phenotyping and used technology like the ones that several people already commented in this thread. (WAS as bad economy left me jobless, but moved to create a business for 3d reconstructions from plants for farmers using cellphone and drone images)

 

The scope of what I'm looking for is too large for me to be pushing it in any one direction beyond tool development.  What you're talking about would be best if I wanted a specific game world converted, not hundreds /  thousands.  My video is intentionally more broad, to try and ignite interest in this, so that as advancements are made on this front in software, people who saw the video can keep this kind of application in mind in the future and get incentivized to make it happen for games they might be interested and hopefully leading to more worlds being explorable. 

 

Additionally, I wouldn't rely on the transformative work defense for distributing these.  While it's possible that could hold up in court, most people wouldn't have the resources to defend against that sort of thing, so it would likely have to be semi-underground.

22 hours ago, Artlav said:

The technique would work well in a game world, but one thing i didn't notice you or anyone mention is that it would take A LOT of time. A game world can be vast, and recreating it via video means you have to cover all the ground, at a distance close enough for the finest details to load in. For a game that lets you drive all over a continent to scale this would be an untractable job.

 

I think you're underestimating the lengths fans are willing to go to in order to "survey" the game world.  I personally would go through some games to get them converted, or at least some levels (I'd love to have Blood Omen 2 converted for example).  Additionally, if someone went through, but missed a few nooks and crannies, if the information was public, someone else could fill those in.  That was kind of the point of the articles I mentioned, people have spent 10 years recreating middle earth, I think others might be willing to really do the legwork on converting the games if they had the right tools and it was well known what to do.

 

For something like an enormous driving game, you would need smarter tools so only SOME areas need to be gone through at fine detail, then have an additional AI to look for pattern recognition.  If you have a game with a hundred miles of road, the road texture is going to be repeated an immense amount of times, you would need an AI anticipating that.  Same goes for other props like trees, houses, powerlines, etc. that are all being re-used.  You would need the AI aggressively guessing what is a duplicate of existing assets.

 

On 1/12/2021 at 4:42 PM, Exnihilator said:

Could video game capture be one of those best case scenarios

Well, yeah.

95% of the math involved go towards deriving the camera position and movement from how the images change, while simultaneously mapping the environment to have something to do the derivation against.

SLAM, simultaneous localization and mapping.

 

If you could precisely control the camera position from the game, you would instantly simplify the problem to a mere mapping exercise of depth and texture extraction.

That won't cover games that don't give you direct control over the camera, however.

 

Ideally you would just read the camera position from game's memory, but if you can access enough of the game data to get the camera position, then you likely can access enough to extract the whole visible scene as raw geometry, rendering the whole approach pointless.

 

I've never made or played online games, so i can't comment on how practical that would be in case of cheating countermeasures.

 

20 hours ago, Ross Scott said:

I think you're underestimating the lengths fans are willing to go to in order to "survey" the game world.

I guess i do. Then again, the larger game worlds i can think of are procedurally generated infinite maps of one kind or another, where you would only need to extract enough resources to get the tune of it to work.

For more interesting artistic worlds, what man can do man can undo.

 

This does kinda loop back to the original problem of only the popular games getting the treatment since only they would have enough fans to do the work. Is there a direct corelation between the scale of the game, it's goodness and it's popularity?

 

 

On 1/13/2021 at 5:46 AM, Ross Scott said:

The scope of what I'm looking for is too large for me to be pushing it in any one direction beyond tool development.  What you're talking about would be best if I wanted a specific game world converted, not hundreds /  thousands.  My video is intentionally more broad, to try and ignite interest in this, so that as advancements are made on this front in software, people who saw the video can keep this kind of application in mind in the future and get incentivized to make it happen for games they might be interested and hopefully leading to more worlds being explorable. 

 

Additionally, I wouldn't rely on the transformative work defense for distributing these.  While it's possible that could hold up in court, most people wouldn't have the resources to defend against that sort of thing, so it would likely have to be semi-underground.

I think you're underestimating the lengths fans are willing to go to in order to "survey" the game world.  I personally would go through some games to get them converted, or at least some levels (I'd love to have Blood Omen 2 converted for example).  Additionally, if someone went through, but missed a few nooks and crannies, if the information was public, someone else could fill those in.  That was kind of the point of the articles I mentioned, people have spent 10 years recreating middle earth, I think others might be willing to really do the legwork on converting the games if they had the right tools and it was well known what to do.

 

For something like an enormous driving game, you would need smarter tools so only SOME areas need to be gone through at fine detail, then have an additional AI to look for pattern recognition.  If you have a game with a hundred miles of road, the road texture is going to be repeated an immense amount of times, you would need an AI anticipating that.  Same goes for other props like trees, houses, powerlines, etc. that are all being re-used.  You would need the AI aggressively guessing what is a duplicate of existing assets.

 

If you're playing a driving game set in Ohio for too long, the AI will eventually start generating heads on spikes lining the road and lava pits cooking the souls of the dammed. 

1 hour ago, Im_CIA said:

If you're playing a driving game set in Ohio for too long, the AI will eventually start generating heads on spikes lining the road and lava pits cooking the souls of the dammed. 

 

I just drove through Ohio the other day. I didn't see any of that, but I didn't get too far off the interstate either so maybe that's the reason why.

 

Would have been a great sight to break up the monotony.

100% on board with idea.

 

It could be I haven't heard of it, but I've wondered for a while now why there isn't a community built game engine produced in a manner similar to Linux.  A community owned and developed engine that was constantly evolving to incorporate the wants of the community rather than budgetary shackles of shareholders.

 

I admit, it would have to be a baller engine to get programmers to opt into it.

On 1/13/2021 at 4:46 AM, Ross Scott said:

The scope of what I'm looking for is too large for me to be pushing it in any one direction beyond tool development.  What you're talking about would be best if I wanted a specific game world converted, not hundreds /  thousands.  My video is intentionally more broad, to try and ignite interest in this, so that as advancements are made on this front in software, people who saw the video can keep this kind of application in mind in the future and get incentivized to make it happen for games they might be interested and hopefully leading to more worlds being explorable. 

This is true, however there are real world limitations. Some reconstruction projects for real cities have taken tens, if not hundreds of hours, and this is due to real world limitations with gpu. We are still not in that level with hardware.

My point is that, if something like this wants to be pushed, we have to start somewhere don't we?

I say, pick and old game you really like. Old too to avoid legalities. Bring the community together to take screenshots of everything randomly, have 2-3 guys that know their shit stitch and clean, and make meshes public.

Then have people explore those worlds.

BAM

Public interest rises, many more projects advance trying to automate this.

 

Edit: i believe i am failing at making a point clear so here we go. Making software to extract 3d data from games using sfm (structure from motion), is NOT a commercially viable option, at least from a AAA perspective.

I don't want to look as if I am shilling my shit, but it is an example.

I made an online application for sfm, which is supposed to let agricultural sectors with less resources analyze crops with just their cellphone. Analize 3d reconstructions from plants, with a toolset for morphological analysis.

 

It was a bitch to get initial funds, and we barely were able to get a webapp going.

I then opened a go fund me here, with no real success: https://gogetfunding.com/plarepi-localization/

Sure, we are only looking to localize and at best, add AI algoritms, the product is already finished... but we are still from a small country and little budget. Getting users in our country has been hard.

 

Reality is, empowering people with tools is a risky venture for it requires as much effort educating people as it does selling said tools. It becomes niche as hell.

It may be because agriculture isn't innovative on its own, it may be our country, and also that we are huge nerds with no social pull... but it doesn't take from the fact that a AAA companies aren't gonna take this risk.

This is why i say that something like "the dream software" must be financed by the fans, for the fans. I don't see it happening otherwise.

Even things like testing, must start with people's help.

Sure, i can make a gofundme project, but, we aren't gonna achieve shit without LOTS of support.

Oh, and those wondering, it should be financed, but the final product MUST be open source.