The software of my dreams

My dream software is an airplane throttle that increases the money in my bank account

The very complex task you asking for, Ross. Cool video, I like to watch your videos about your ideas.

I think it will be a great idea for games you can not access the data, like yea, games on encrypted servers. Fear that with all this "google stadia" stuff there might be even more of them. Something is better than nothing, you are right.

However in general game worlds and game looks and feels is very complex thing, as you mentioned in the video.

 

World data, external meshes, palettes, shaders, collision. The first thing I had in mind is my internal video game purist screaming inside my head. I think Daggerfall unity was an awesome example. I like that this port exists, that Daggerfall got back its modding community, however, I love the original engine, Bethesda's Xngine. How it feels, how it renders things, and how a lot of things worked in the original game. And this port on unity is not... very accurate. Using default unity stuff, it making lower resolutions and original palette only as some post-shader "emulations". Walking physics, rain and snow effects, headbobbing effects were different, and just some unity stuff that looks weird. Speaking of drawing distance it also has limitations, and more than that, terrain generation is inaccurate. It just using the original height map of the game, but all magic done now by unity instruments, so you have now this smooth terrain with huge view distance that looks not much like the original. This game feels like a "unity game with Daggerfall assets" and I don't like this feeling. Sad that XLengine project died, I hope Lucius will come back for it after he will end with "The Force Engine", my favorite kind of projects of new engines is ones that reverse-engineer original as closely as possible, with new features added only something on top on the accurate representation of the original game.

And problem of such projects not lack of data usually, everything cam be reverse-engenereed, just lack of enthusiasm or interest in some games. There are huge amount of games that will benefited from ports, patches, improvments but there are no one really interested to doing something complex with them.

So yea, when we take the idea of "ripping off worlds, assets, games, using our or machine's perception" it will end in a very inaccurate approximation even if the software is kind of perfect. It it something that will be better used for capturing real world, thank god physics laws generally the same, than for worlds created by man. For games (at least, to which files we have acces to) this dream software for me is instruments like Ghidra, that allow to reverse compiled product back into code.

 

However, speaking about your ideas about "video game musem" this is absoluetly awesome idea! You know, long time ago when I played Garry's Mod I had wierd idea what if porting or making model for everything, port here all props, models, maps from a lot of games, and more, universes! This something what can be started before dream tool you are discribing, but might leed to legal issues. However for most older games I don't think anyone would care lol.

 

Addition: this probably will go off topic but I wondered about the game that has all the cars ever made. Or, at least, a lot and updating them. I'm not a super racing gaming fan, in my dream game I want just to go to the garage, spawn a car from a database, for example, a 1986 Opel Kadett E sedan with red color and 1.4 engine. Or, 1984 Ford ltd lx, or some 1909 De Dion-Bouton Torpedo. Be able to disassemble it in VR, assemble it back, ride it around with real characteristics, crash and repair. Wierd idea that haunts me for years, really.

4 minutes ago, ultrayoba said:

The very complex task you asking for, Ross. Cool video, I like to watch your videos about your ideas.

I think it will be a great idea for games you can not access the data, like yea, games on encrypted servers. Fear that with all this "google stadia" stuff there might be even more of them. Something is better than nothing, you are right.

However in general game worlds and game looks and feels is very complex thing, as you mentioned in the video.

 

World data, external meshes, palettes, shaders, collision. The first thing I had in mind is my internal video game purist screaming inside my head. I think Daggerfall unity was an awesome example. I like that this port exists, that Daggerfall got back its modding community, however, I love the original engine, Bethesda's Xngine. How it feels, how it renders things, and how a lot of things worked in the original game. And this port on unity is not... very accurate. Using default unity stuff, it making lower resolutions and original palette only as some post-shader "emulations". Walking physics, rain and snow effects, headbobbing effects were different, and just some unity stuff that looks weird. Speaking of drawing distance it also has limitations, and more than that, terrain generation is inaccurate. It just using the original height map of the game, but all magic done now by unity instruments, so you have now this smooth terrain with huge view distance that looks not much like the original. This game feels like a "unity game with Daggerfall assets" and I don't like this feeling. Sad that XLengine project died, I hope Lucius will come back for it after he will end with "The Force Engine", my favorite kind of projects of new engines is ones that reverse-engineer original as closely as possible, with new features added only something on top on the accurate representation of the original game.

And problem of such projects not lack of data usually, everything cam be reverse-engenereed, just lack of enthusiasm or interest in some games. There are huge amount of games that will benefited from ports, patches, improvments but there are no one really interested to doing something complex with them.

 

However, speaking about your ideas about "video game musem" this is absoluetly awesome idea! You know, long time ago when I played Garry's Mod I had wierd idea what if porting or making model for everything, port here all props, models, maps from a lot of games, and more, universes! This something what can be started before dream tool you are discribing, but might leed to legal issues. However for most older games I don't think anyone would care lol.

 

So Ross's dream software is basically one of the cornerstones for a Holodeck? Nice.

>you can only play the game in polish if you are in poland

Jesus Christ EA, what are you even doing?

There is research for this on small scale real life locations: https://nerf-w.github.io/ can't find code implemented for nerf-w but vanilla nerf still exists: https://github.com/yenchenlin/nerf-pytorch might require a lot of compute though. And before anyone asks, the 3d models can be extracted from it. It requires marching along the rays. Here is an anime girl-guy doing a toturial on it:

 

(Last edit promise) more info on methods that probably fit better can be found here: https://paperswithcode.com/task/3d-scene-reconstruction#code

Hey @Ross Scott about SA2 being ported into SA1, it's probably because SA1 controls differently and there are a lot of people that prefer how it handles compared to SA2

Ross, I hope you realize that even if this software were possible to make, it would be completely illegal. It sounds to me basically like a tool designed to make stealing copyrighted material as easy and efficient as possible. 

2 minutes ago, daisekihan said:

Ross, I hope you realize that even if this software were possible to make, it would be completely illegal. It sounds to me basically like a tool designed to make stealing copyrighted material as easy and efficient as possible. 

That was never an issue before 

5 minutes ago, daisekihan said:

Ross, I hope you realize that even if this software were possible to make, it would be completely illegal. It sounds to me basically like a tool designed to make stealing copyrighted material as easy and efficient as possible. 

This sounds like one of those comments someone makes before they finish the video.

1 hour ago, ultrayoba said:

World data, external meshes, palettes, shaders, collision. The first thing I had in mind is my internal video game purist screaming inside my head.

Yeah, this is NOT going to do much for purists, that's why I added the "v. nothing" example in the video.

 

As for every vehicle, you almost just need a driving games that highly open to modding that a lot of fans coalesce around.  No publisher is going to get the rights to show all vehicles, but you could have fans creating mods for tons of different ones.

2 minutes ago, daisekihan said:

Ross, I hope you realize that even if this software were possible to make, it would be completely illegal. It sounds to me basically like a tool designed to make stealing copyrighted material as easy and efficient as possible. 

To be fair, how would you prove it is intended to take copyrighted materials? It's the same deal as with the console emulators. Making them and using them is not a crime as it is useless without a game's .ISO, it's only once you pirate the console game's disc image it can be proven as a crime.

Nothing wrong with making or using a program to access your GPU (which you own) to take parts from it's memory or whatever while playing game (which you also own) and converting that data into a different format. It's only a crime once you share this data online and it's clearly recognized as a part of intellectual property that you do not own.

2 minutes ago, Konstantin3001 said:

To be fair, how would you prove it is intended to take copyrighted materials? It's the same deal as with the console emulators. Making them and using them is not a crime as it is useless without a game's .ISO, it's only once you pirate the console game's disc image it can be proven as a crime.

Nothing wrong with making or using a program to access your GPU (which you own) to take parts from it's memory or whatever while playing game (which you also own) and converting that data into a different format. It's only a crime once you share this data online and it's clearly recognized as a part of intellectual property that you do not own.

Yeah, let's call it the Plex loophole 

This video is a perfect example of why I'm in Ross's fanbase. I don't care whatsoever about the topic itself, but watching Ross's eyes burning with passion just energizes me in a certain way. Seeing someone so zealous - no matter about what - kinda awakens me from slumber and motivates me to pull my own shit together. I'm sure there is a specific term in psychology for that.

 

(It's also exactly the reason why I'm not crazy about FM. It just doesn't have that... vibe I was talking about).

Speaking of streamlining the process as much as possible, the creator of Garry's Mod is currently making the S&box, with the original vision of being able to just "stick your fbx and png files in your addon and load them and it'd all be good" or at the very least making content for Source 2 as easy as possible. That's kind of on the other side of the dream process from the video, but the S&box project could be an easier solution for the people then working with unreal and unity to create worlds and have a native VR support for them (being based on HLAlyx and all).

yknow one of the things i would love about extreme VR tech (the treadmill thing ross was talking about) is how it would bring fourth a new form of art. that discussion topic of "are videogames art"? well putting on a VR headset would be something new and awesome. it wouldnt be a game just drink in someones cool vision of an environment.

1 hour ago, Chrisfratz said:

Hey @Ross Scott about SA2 being ported into SA1, it's probably because SA1 controls differently and there are a lot of people that prefer how it handles compared to SA2

I mentioned in the youtube comments, but I think it's for a couple reasons. One is that Tails and Gamma in SA1 are way better than the slow mechs in 2. There's mods to correct the turning and increase the jump height a little bit, but the SA1 shooting levels are so much faster and more fluid than anything in the second game, and you don't have crap like Eternal Engine blindsighting you from behind walls. Plus, even if you use his no-mech mode from the chao gardens, Tails can't spin jump and his flying only hovers. Secondly, SA1's palette lighting system (called the Lantern engine) was completely reverse-engineered by fans to work in the crappy PC version, and now is a standard part of most modpacks, and looks way cooler than the flat Adventure 2 lighting. One look at Casinopolis is all you need.

 

The final reason is that Sonic fans are obsessed with putting Sonic games in other Sonic games. Nearly every 3D game before Generations has had most of its levels ported into Generations at some point. Getting that to work was actually a huge technical achievement, since Unleashed runs on the same engine but its stages were never playable on PC due to it being the only console-exclusive 3D game that you still can't emulate properly (RPCS3 is pretty close, but it definitely wasn't in 2013). Even still, you have 06 ported to SA2, Heroes ported to SA1, Black Knight and Colors in Gens, Unwiished in SA2, even Lost World has a significant number of level ports, and that game is the Sonic version of wearing pants that go all the way up to your nipples. Hell, people have put Generations levels back into Unleashed. It's just a weird pathological obsession the community has, probably because all of the pieces for the perfect 3D Sonic engine are technically in our hands, but they're just not quite put together in the right way to work. I think I remember melpontro and SF94 working on something, and that Hero engine looks pretty neat, but it's still gonna be a while.

The other thing that might be of tremendous help is a universal unpacker/converter. If you have the archives with models, textures and animations, getting resources would be easier then ever! It would do nothing for streaming only games, but if someone saved the files from Darkspore or other "game bricks", the assets can be saved, or at least salvaged and repurposed in the different engine. It's actually way more realistic and may have better results then a Neural network powered software trained to work with recordings or a program for ripping 3D data from GPU, due to the problems discussed in the video. But in cases of streaming only games, games with 3D data handled by server and games with weird archiving/encryption/formatting the unpacker/converter can't handle, those two might be the only chance, yeah.

Ross, I want to say that your wish is somewhat true already, because of a little thing that exists called photogrammetry.

 

Basically you take a bunch of images with similar features, dump 'em into the software of your choice, and you get a mesh complete with textures!

 

This is not perfect however, and does suffer from some of the problems mentioned in the video, I.E. lighting data essentially being baked in, struggling with reflective surfaces, etc. With that said, if your images are good, you can get pretty good results (you may have to clean up the mesh in selective cases though).

 

I'll upload some examples if I get the chance.

 

Anyways, as for what software to use, I'd personally recommend using Meshroom, as it's what I've used: https://alicevision.org/#meshroom

Also, it's completely free, open source, and stupidly easy to figure out and use. No money required. (Edit: Forgot to mention this, but in my testing, "live reconstruction" seems to be broken in the 2020 version. The software still works though.)

 

Another program I know of, but haven't used, is called RealityCapture: https://www.capturingreality.com/

All I really know about it is that Valve have used it, and they have a free version where you pay if you're satisfied with your results, and want to use 'em.

 

Oh my gooood. Talking about walking through worlds reminded me of a tool on GitHub to inject support VR into Unity games. This app only works for a few versions of Unity, hasn't been updated in a few years, and doesn't care that a game wasn't made for VR; but it's breathtaking to be standing there in the middle of a running game when it works.

 

 

Hopefully somebody can poke at the source code and find ways to expand it.

This also looks promising https://github.com/magicleap/Atlas