Do you have the technical knowledge to conduct an audit of an analytic dashboard? If not, you should go get it as soon as possible. As of February 2017, all the most used audit standards allow us to bring specialists into our audit teams to train and enhance our capabilities. If your team finds itself in need of some technical knowledge, find an expert and learn from them!
The process of turning data into information to present it in a simple manner can be incredibly complex. I believe this irony is primarily because most available data is not formatted for analysis. Building a large, custom data set with the exact list of features you desire to analyze (Design of Experiments) can be very expensive. If you have pockets as deep as big Pharma or are ready to dedicate years to a PhD, it’s definitely a great way to go.
Our last blog on trusting data analytics explored how the industry practice of “data cleaning” can spoil the reliability of an entire analysis. But problems can also occur with perfect, clean, complete, and reliable data. In this post we will explore the topic of data provenance and how the complexities of data storage can sabotage your data analytics.
The truth is… business data is structured and formatted for business operations and efficient storage. Observations are usually:
- Recorded when it is convenient to do so, resulting in time increments that may not represent the events we actually want to measure;
- Structured efficiently for databases to store and recall, resulting in information on real world events being shattered across multiple tables and systems; and
- Described according to the IT departments’ naming conventions, resulting in the need to translate each coded observation;
Lack of trust in source data is a common concern with data analytic solutions. A friend of mine is a product manager for a large software company that uses analytics for insights into product sales. He told me the first thing executives and managers do when new analytic products are released in his NYSE-traded, multi-billion dollar company is… manually recalculate key metrics. Why would a busy manager or executive spend valuable time opening up a spreadsheet to recalculate a metric? Because he or she has been burned before by unreliable calculations.
I’ve been exploring the subject of unreliable data since a recent survey of CEOs revealed that only 1/3 trust their data analytics. I have also been studying for an exam next week to earn a Certified Analytics Professional designation to formalize my knowledge on the subject. While studying each step in the analytics process on INFORMS’ analytic process, the sponsoring organization for the Certified Analytics Professional exam, I’ve considered how things could go wrong and result in an unreliable outcome. In the flavor of Lean process improvement (an area I specialized earlier in my career), I pulled those potential pitfalls together in a fishbone diagram:
While giving a presentation on Analytics during a recent event, one of the meeting participants asked how the Audit industry felt about data products created using Analytic processes. On first thought, I consider Analytics to be a form of “analytical procedures”. This was my response but I had to qualify it by acknowledging that I wasn’t sure how different auditing standards addressed the topic. Over the last few days I’ve been able to do some research and pull together a quick synopsis of how the most commonly used Audit standards define the work behind Analytics. In summary my initial impression was pretty close… several of major Audit standards define this type of work and emphasize the reliability of data that underpin Analytic data products.
This post may disturb some “old school” auditors. In fact, used to be a self described old school auditor. If I couldn’t find a work paper in the permanent file, someone was going to get an earful about their ability to keep reliable documentation. But the business sector has evolved at a frightening rate since those days. Surprisingly, many audit professionals still consider sampling and testing to be their go-to procedure.
I’m not knocking the old student’s t-test. It’s still the right tool for some situations. But it’s been harder and harder for me to find those situations in recent years. Most of the subjects I audit now can easily be scrutinized using data mining, or even scripted into an automated monitoring report. Continue reading “Quit Sampling!”
While trying to get this blog to appear higher in internet searches, I ran into an interesting article on the Journal of Accountancy discussing Analytic Auditing. The article is written from an external audit perspective and focuses on two benefits. First analytics provides auditors with greater insights into their clients’ business that help to quickly get up to speed on the external audit customer’s business model. Also the article mentions how analytics provide better service to clients.
The article also states a position that I’ve had since learning analytics as an external auditor… that external auditors’ use of analytics lags far behind that of internal auditors. I think a key reason for this is access to and familiarity with data. As an external auditor it took several weeks for me to gain access to a new client system. Once the client granted my access, I didn’t have much time to pull something useful together. Rarely did my projects have more than one or two models. As an internal auditor, I’ve had the same difficulties getting initial access to the system. But once granted access I can continue to develop models as long as my results are useful to the organization. On certain projects/systems this period lasted for several years and allowed for deep exploration and understanding.
To me one of the most impactful sentences from this article is “The profession [external auditing] needs to achieve a “quantum leap” to redesign audit processes using today’s technology, rather than using information technology to computerize legacy audit plans and procedures. ”