How much data storage is necessary to effectively monitor and troubleshoot application performance?
“Problems only happen when you’re not watching”
is an old troubleshooting proverb that helped drive our thinking. How often have you looked over a co-workers shoulder as they exclaim, “This was slow 5 minutes ago, but it’s not now!” Indeed, reproducible application problems tend to be solved quickly after they’re reported. It’s the intermittent problems that linger. This is not because the underlying issues are more difficult to fix, but rather due to a lack of data in nearly every case. When we can reproduce the problem, we can also generate the detailed information to understand and resolve the issue. When we cannot reproduce it, we’re left with whatever metrics were collected by our monitoring tools. Depending on the level of detail that these tools choose to store, that data may be insufficient.
Some argue that storing this “High Definition” data is unnecessary; we disagree.
Many APM products record metrics, but throw away the transactional data that those metrics are based on. If 99.9% of the transactions will never be reviewed, why bother capturing them? That would be a fair argument if it weren’t for the fact that the other 0.1% of the transactions is so extremely valuable. Rather than repeatedly identifying the same problem and trying hypothetical solutions based upon partial data, you can find the root cause and fix it correctly the first time if you have the right data to act on.
By making all of the transactions searchable, we can find answers to questions that we never thought of in advance.
Each person may have unique questions about an application’s performance. Many of these will be captured by pre-calculated metrics, but not always. For example, “What are the IP addresses of readers of this blog post who are running Android 4.0 and had a DNS resolution time > 2 seconds?”
We believe IT needs easy access to all transactions all the time to maximize the value they provide to lines of business.
Ultimately, this is why we made the ambitious choice to store all transactions all the time for our AppResponse Xpert SaaS solution, in line with prior design decisions we made for our APM Xpert suite. Whether that data is network packet streams over 10 Gbps networks, code-level transaction traces on dozens of virtualized systems supporting a production application, or end-user page hit records for a global commerce application hosted in “the cloud,” IT needs easy access.
Does the cost of storing so much data outweigh the benefit?
Not if done properly! Many industries are grappling with Big Data challenges and the APM business is no different. We worked very hard over many years to build the proprietary NoSQL database technology that enables the OPNET APM Xpert suite to conquer the Big Data problem for APM use cases. Combined with analytics for automating the problem-solving process,
The result of storing every transaction is faster answers to the hard questions.
Article provided by OPNET – Application Performance Matters