Almost every regulation and standard has logging requirements. This requirement is there to ensure that there is data that can show attacks and breaches. But there have been numerous breaches in which the the breach happened over a period of time and they were not discovered for a long time after.
Most of the organizations that have been breached, as well as others, have logging mechanisms in place. There are logs that collect huge amounts of information starting all the way from someone hitting the web server to someone modifying something on an internal server. With all these, why are breaches still difficult to unearth in time? The problem may be that there is too much information. If you look at the logs from a single application and try to track say, a bug, you will have to wade through pages and pages of log entries. Just imagine going through all the logs for all the applications and network resources, collating them and trying to find patterns of behavior that can indicate attacks.
This can be a huge task in the best of circumstances. But in these tough economic times, layoffs have become the norm. Almost every organization has to make do with lesser number of staff and still keep up with all the things that need to be done. Experienced staff may not want to sit and analyze gigabytes of log entries. While some of the stuff can be automated, there is still a requirement for human intelligence to make the correct calls. What ends up happening is that there is no proper review of logs that can indicate an attack or breach.
Another reason is that some organizations do not have proper processes that define what needs to be done. More often then not, they do, but these processes are not followed. The graph below shows some of the reasons for breaches not being identified promptly.
The primary reason is that there is too much data from a lot of resources. Even taking a very simple case, we are looking analyzing logs from the network firewall, web server, app server, application and database logs. Each of these logs can be in different locations and they may be in different formats. Getting all the data together in a way that provides a complete picture is the challenge that gets the better of most organizations.