An artist's depiction of an Extended Unit Death
Well hard is relative when we're talking about games that have million dollar budgets. Also I'm not buying that. The foundation is already there, as pointed out by LL.
LoveLess described a decision tree, which is arguably not even qualifying as AI because it doesn't learn. The only thing we have for actual AI is IBM's Watson, and as far as I know, the supercomputer has not been integrated into any games. So I would disagree that the "groundwork" is already there, ready to be utilized by anyone.
Are we talking about true AI here, or just decision tree-based analysis in game theory? Because the former is hard, and the latter is certainly not trivial for open environments where there are infinite* different states to consider.
The more flexible and comprehensive a game is, the more complicated the AI has to be, and anything other than a perfect decision tree covering almost every scenario is going to leave notable flaws that make the AI easy to defeat. The first Halo game had a limited number of weapons and vehicles per campaign level, whereas later games introduced more complex variables such as dual-wielding and more destructible environments. The behavior of enemies in the later games haven't gotten worse from what I can tell: they still flee, hide, and otherwise react to the player (in fact, I think they've improved in this regard).
Now, here's the primary reason why AI has seemed to get worse in modern games: a lot of older games would have scripted scenarios that work ideally for the campaign level they were designed for (typically against only one player), whereas modern games use a generalized AI to handle any scenario/map against multiple enemies/players. Sacrieur brought up BWAPI, which is a perfect demonstration of these two approaches: the AI that try to cover all scenarios to compete against various builds perform poorly against ones that bank on a specific strategy that would easily be thwarted if applied outside the environment it was designed for (e.g., against a human opponent). Arguably more work goes into designing the generic AI, but the success ratio is way lower than the AI designed for the scenario of taking down other AI.
So we come down to two differently designed AIs: those created for specific campaigns/scenarios, and those designed to perform in any scenario. Your concern seems to be that most developers opt for the latter approach, which means the AI doesn't perform optimally in many situations. The latter approach is the
only option for games that allow for custom maps with AI computers, and it's the only approach if you want an AI that you can reuse on sequel games or expansions. The former approach (hard coding) will make an extremely competent, well-polished AI for the map it was designed to handle, but that AI would be completely worthless in any other scenario (and forget having support for custom content with AI designed this way).
Why don't developers design for both? Well, because AI is hard. Development time is expensive, and having one AI is cheaper than having two, and it's not exactly a feature you can market to audiences to take both scenarios into account. That's not to stay there aren't hybrids: even StarCraft, for example, has a generic AI, but also scripts for various campaigns. But guess where that leaves the impression of the AI outside of the campaigns?
TL;DR: The smartest generalized AI will not outperform the dumbest hard-coded AI.
*Arbitrarily large but finite
Post has been edited 1 time(s), last time on Jun 24 2014, 8:55 pm by Roy. Reason: Fixing typos made from my phone