We face this kind of issue everyday. There are two different approaches to web application penetration tests:
- An increasingly number of companies are buying automatic web scanners, run them, generate some results and put them in a report-shaped tin, ready to go to the client. No human interaction with the application is needed.
- Some other companies allocate X numbers of days of a highly skilled consultant to assess the security of your web application. Among many other tests the consultant will also run automatic web scanners, but that is only scratching the surface of a real penetration test. The consultant will use all his/her experience to analyse many other factors of the application.
Penetration testing is all about assurance. In the first case the client will get some useful results, no doubt about it, but what level of assurance is it going to get? The report will cover the vulnerabilities discovered by XYZ software. Is that enough? I don’t think so, but that is for the client to decide. There is no question that the report will be incomplete and many issues will be missed.
In the second scenario the client can get the assurance that the results obtained were the work of a motivated attacker focused on the application security for X numbers of days. Is that enough? Again, it is up to the client to decide but in my opinion it gets so much closer to an acceptable assurance level.
It all depends on what do you want to be protected against. The decision in yours.
