As for the cases of failure, the answer is overwhelmingly, "of course it didn’t work; the project didn’t have enough good people."
This is of course followed with:
Good people can build good software no matter what methodology they use.
I think though that one of the things they miss that agile gives you the opportunity to do is fail fast. This is a tremendous advantage in helping predict paths features that customers won't find useful and to also determine when a particular technical path is the wrong way. But knowing this, if you decided you need to scale or need a platform, you are talking now of a major undertaking that won't really fit in the "Agile" world.
So if I were to say what the best software development process would be, I have to say that it has to be something that results in quick iterations for the design and experiment phase which would be followed by a planning process to convert this prototype into something becomes maintainable and meets more rigorous production requirements. Maintainability is key. Along with good documentation.