Heh!

First of the link to The Great Pyramid of Agile. This article was fairly amusing. Covers many of the issues of good platforms/architectures. Also takes into account that agile allows for many deviations from the original plan (basically an allowable form of scope-creep if you ask me).

As for the cases of failure, the answer is overwhelmingly, "of course it didn’t work; the project didn’t have enough good people."

This is of course followed with:

Good people can build good software no matter what methodology they use.

I think though that one of the things they miss that agile gives you the opportunity to do is fail fast. This is a tremendous advantage in helping predict paths features that customers won't find useful and to also determine when a particular technical path is the wrong way. But knowing this, if you decided you need to scale or need a platform, you are talking now of a major undertaking that won't really fit in the "Agile" world.

So if I were to say what the best software development process would be, I have to say that it has to be something that results in quick iterations for the design and experiment phase which would be followed by a planning process to convert this prototype into something becomes maintainable and meets more rigorous production requirements. Maintainability is key. Along with good documentation.