Edit History

On a more general note, of why AI did not advance much in the past years, I think that it is simply a problem of indies not having enough resources to develop something better and the bigger companies not having a reason to do it. As already mentioned it is a job for them, games with poor AI are still selling, so why bother improving it when improving graphics is easier.

Honestly, I don't remember the AI in F.E.A.R. being notably smarter. The simulated radio chat was nice and I remember them flanking me once but most of the time it didn't seem different than most shooters. But sure, in side by side comparison with other games it would come on top. One of the reasons it didn't seem so good could also be that the map layout did not have that many tactical options.
I guess the best when it comes to AI in gaming now comes from AIs developed to work as a player. Like all those Starcraft playing AIs and more recently MOBA AIs. I think I also remember someone boasting about an neural network based AI, though NNs are basically just good old throwing more hardware at the problem.

With the hardware we have today there is an opportunity to grab some graphicaly undemanding engine and use that spare power to create something really ambitious, like those games which simulated whole space systems with economy and NPCs traveling around but even bigger. I would like to see that. A capable AI squadmate would be cool too. Or a believable stealth game.