on the risk of inaccurate 'assessments'

I've pondered on a '$security_topic is dead' title for this blogpost, but I managed to steer clear of that one. I personally don't believe that anything (except for Antivirus ;-)) is really dead and my buddhist little toe tells me that if anything is dead, it will most probably live on in another shape or form.

I've been involved in penetration tests, security assessments and audits of different kinds (both regulatory and not) and from both perspectives (as the tester and as the testee). When sitting on the tester chair, I've experienced how hard it is to translate ones findings to a proper report that, without resorting to FUD, accurately assesses the risks the customer is exposed to. On the other hand, I've been frustrated with numerous reports I received that qualified risks as High, Medium, Low and/or Red, Yellow, Green. From a customer perspective, what am I to do with these 'values' ?

While a qualitative assessment is the easiest way to qualify risks, it also completely disconnects us from the business and/or the customer. When making a qualitative assessment we are not taking in account the nature of the business and the processes that our customer actually practices to run his business.
Some practicioners refer to 'best practices' or 'good practices' (marketeers, please take a one-way ticket to a deserted island ?) but still I don't feel that this positively impacts the result of the analysis.

Within the limits of a penetration test, quantitative risk assessment is nearly impossible. First and foremost because you will never* receive accurate numbers within the limited timeframe but again also, and more importantly, because as a technical tester you are completely and utterly disconnected from the business.
Running meaningless numbers through complicated formulas and creating scatter plot
graphs representing risks are probably comparable to trying to kill a deer by
throwing a bullet at it. It does not work.

In short :
a) penetration tests and security assessments are, today, mostly technology oriented.
Yes, we do assessment on the process level too, but not as much and not as thorough.
b) results are often poorly communicated due to lack of connection with the business and/or lack of feedback from the business.
c) customers are not up to par considering risk assessment as a vital part of doing business. Security is still the responsibility of IT.

Conclusion :
If we want to create value by providing penetration testing and security assessment services, we should stop selling 5 days, fixed prices 'solutions' providing a detailed report. We should engage with our customer on a very high level so we can first understand the business and then tailor security solutions to their needs by going through shorter iterative cycles solving problems one at a time, raising awareness throughout the business and in the end providing a company with the necessary processes to tackle security processes on their own.

I'm looking forward to be a part of this in 2009.