Many organizations conduct regular penetration tests of their applications and infrastructure. Actually, this is required by certain industry standards such as the PCI DSS. The problem with penetration tests is that it provides a very one dimensional view of the landscape and may not bring out certain underlying issues.
As a consultant I have performed quite a few application penetration tests and vulnerability assessments. In almost every case, the lack of findings was taken as evidence that the application was secure. There is a very significant difference between the fact that no vulnerabilities were uncovered and that the required controls were in place.
There can be instances where there are no findings when the controls are not in place or they are not implemented properly. A lot of people think that if they performed a penetration test of their application and did not find any vulnerabilities, their applications are secure and unbreakable.
Some important factors play a very significant role in the effectiveness of penetration tests. The most important one is the skill of the pen-tester. Obviously, the level of skill and knowledge that the pen-tester has is directly proportional to the chances of finding vulnerabilities. Security professionals come with varying degrees of skill and there are a lot more that just rely on automated tools than there are ones that know what they are doing.
The next factor is how the application is scoped for the pentest. When testing large enterprise apps, the boundaries may have to be defined to include only a part of the application or a subset of applications due to budget, time and other constraints. In these days of multi-layered applications, the effects of a successful test may not be apparent within the boundaries of the test.
On quite a few occasions, I have had to go outside the boundaries to another application, that used the data stored by the application in scope for the test, to verify if there were any vulnerabilities.
One more thing to consider is the fact that most of these tests are performed in a non-production environment. And In a lot of these cases, these environments do not mirror production environments. The differences in application configuration or environment can impact the results of the test. Performing a penetration test on production servers will require some of the tests (that can bring down the servers) to not be included and is not recommended, except against non-critical servers.
While penetration tests definitely have a role to play, it is not advisable to rely solely on them to verify the security of your applications. Threat modeling is something that will help identify issues long before they manifest themselves in code. But that is a topic for another post.