An intrusion, in an outsourcing data center, that works in spite of security

I work in a data center, focusing network connectivity and network security. Solution architecture centers on providing the layered security promoted by the industry in general and SANS in particular. We strive to incorporate the highest level of security and performance permitted by the customer, budget, and availability of stable, suitable technologies. We buy or receive what is agreed fits the solution and implement it in a way that it will perform well. After implementing the environment, the next step is to monitor the individual components of the solution and keep up with required changes to the network components (intrusions, patches, new versions and technologies.) Periodically we, or a third party, with the approval of the customer, will attempt intrusions or perform simple audits of the environment. This approach to security, typically effective and reproducible, combined with a fairly constant and high workload, left me with the solid belief that the networks were secure and that we were doing what was needed to continue the trend. However that changed one day when one customer reported that a break-in had occurred.