> but I originally wrote AMD off, since my understanding was they got rid of the option to force AA in games at the driver level on Linux.
I would recommend you try both AMD and Nvidia cards/drivers. Sadly forcing things like AA reliably seems to be outside primary goals of any vendor, so it's hit and miss regardless of the philosophy.
The AMD ones are first-class open source and do not come with a GUI configuration tool. Instead they're compatible with the "standard" open source tools (xrandr, driconf, Xorg.conf, etc) as well as a wide variety of hacks & library replacements (libstrangle, etc).
The Nvidia ones come with a GUI tool, but replace several system libraries so other tools do not work. The GUI tool they come with is basically the _only_ source of configuration you can do. Also note the potential difficulties with older Nvidia cards (you will have to make sure the right [old] version of the Nvidia drivers are installed).
> If the scaling can be done on the card / OS itself and it's downsampling the image to be sent back at native resolution, great.
Yes, that's what xrandr --scale does. The monitor does not even flicker (or notice), it's still being sent whatever resolution it was already set to.
N.B. xrandr is a standard X11/Linux tool for playing with screen resolutions, rotations, arrangements etc. Many GUI monitor configuration programs use it (or the xrandr interfaces) behind the scenes. It's not hard to use or learn, esp compared to many other command-line tools. Will not 100% work with Nvidia drivers.
> I'm not going the custom resolution route that can send rates that the monitor can be incompatible with [...] being good for the monitor
Note 1: not true with LCDs, their mux/decoder flatout rejects out-of-spec signals. Some are more forgiving (eg allow more than a few % timing stretch) but that's about it. Yes a potential problem if you are using a CRT.
Note 2: you won't be able to do this accidentally, adding new modelines to X11/Linux requires long commands full of numbers (and modeline calculator programs (unless you're crazy). Not something to worry about unless you want to.
> Is a Geforce 770 really not enough muscle to run 15-20 year old games on Linux?
The "on Linux" factor is "it depends". Some games run faster under Wine than Win, others visa versa. Some stutter more, some less. Also the variables of whether or not you are using dxvk & what graphics drivers (radeon, amdgpu, nouveu, Nvidia, etc) make a big different. There is a whole stack of libraries between the game and your human peripherals that differs from Windows: wine .dll libs, directx->opengl (wine default) or vulkan (dxvk) shim libs, userspace opengl/vulkan (eg mesa), kernel opengl/vulkan drivers (eg radeon, amdgpu) and of course video card firmware.
I've been playing some Tony Hawks games under Wine on my desktop the past week. THPS 4 has a horribly laggy level even on a good computer. THUG2 runs a lot better. These are games from 2000-2004 that may or may not be able to run faster than period computers because either of the different system stackup or perhaps in-built limitations, I don't have a fast system with period OS (XP) to test on.
> Say I have a DirectX 8 game and a DirectX 9 game that's not in any Linux game database.
That's most of the games I play 
I've got a folder of notes over the years of individual tweaks, methods or manual cracks I've had to implement for some more obscure stuff, I should probably write some of it up.
> What steps would you recommend I take for:
> 1. Getting each game to run
There is no one correct approach to step 1, other than:
1. Try
2. Furiously Google error messages
3. Ask for help on Wine forums or mailing lists if you can't proceed (I've had what I thought were impossible problems answered with simple magic. Summon the gods of the d3dx8->d3dx9 shim dlls :D)
Some of the steps of my approach these days:
1. Create a fresh Wineprefix for each game, so installing arbitrary DLLs does not affect the other games I have. If you're only starting then just ignore this step, learn it later when you start having issues.
2. Try and run and install the game.
3. If something goes wrong: check to see if the game is on WineHQ.
4. If not, google around. Often the fixes required for playing the game on modern Windows also apply.
5. Run the game from the console, so you can see if it's spitting errors about a certain DLL or feature. Often you just need to install some arbitrary windows stuff (d3dx9 libraries, etc). Again, follow the fixes Windows users have to use. Winetricks is a good shortcut program for getting & installing stuff.
6. Pull out the big guns. WINEDEBUG=+dll,+reg wine program.exe and then lots of perusal. Sometimes it's stupid stuff, other times it's multithreaded mayhem and I have no hope of working out what's going on. When I get to the point of disassembling I know I'm lost and crazy.
7. Ask for help in a sqeaky voice on the WineHQ forums. As per above, some nice people there 
I don't use any of the GUI frontends (Lutis now?) so I can't comment on those.
> 2. Attempting to force antialiasing on said games.
Try all of the methods mentioned in this thread. No single method will work everywhere. Every game initialises and uses its graphics api (dx7/8/9/etc, opengl, glide, etc) in completely unique and arbitrary ways, what works for one will not work for another. Things are less standard in the 3D world than you think, remember your custom 3D-in-every game programs?
The xrandr --scale option will "always" work (provided the game can be made to play at the higher resolutions it now thinks your monitor uses) but you will end up with tiny HUD+text unless the game is smart enough to scale these with resolution. It will also potentially be the slowest, but I doubt rasterisation is going to be a bottleneck or FPS limiting factor for old games unless they are in a software-rendering mode.