In my prior blog post, I wrote about different network problems that negatively impact application performance. In this post, I’ll follow up with non-network problems that impact application performance, but for which the network provides a unique vantage point from where such problems can be identified and solved. In the next post, I’ll tie everything together by describing how to determine if the network is at fault and how to get the other organizations to understand more about application performance.
A good Application Performance Management (APM) system identifies clients that have these types of delays. It requires looking at the client-to-server transactions and identifying when the client is paused due to internal processing. The analysis needs to differentiate between the client workstation application pauses and the “think time” of the human who is interacting with the application.
The server teams don’t like to hear it, but the most common causes of slow application performance are the applications or the servers themselves. I’ve found that it frequently is not the network that is the cause, even though the network often gets the blame.
Modern applications are typically deployed on a multi-tiered infrastructure. There often is a front-end web server that talks with an application server. The application server in turn talks with a middleware server that queries one or more database servers for the data it needs. These servers may all talk with DNS servers to look up IP addresses or to map IP addresses back to server names. All it takes is for any one of these servers to have performance problems and the whole application runs slow. Of course, the problem is then one of identifying the slow server out of the set of servers that implement an application.
Understanding the interactions between multiple components in an application is an essential part of understanding the root cause of performance problems. This process, called Application Dependency Mapping, is typically part of an integrated APM approach, and ideally leverages information from already in-place monitoring solutions to draw a dependency map between system components. The network provides a unique vantage point to derive these relationships, and as such the network team can provide strong value to the application and server teams.
Although we can collect a lot of very rich information from the network, using packet capture tools to answer the question of “Is it the network or the application?” could take many, many hours of work. All the while, the application is running slow, affecting the productivity of anyone using that application.
I’ve used Application Response Xpert to significantly reduce the time to identify why a slow application was slow. Once you have set up the proper monitoring points and some basic configurations, it is very easy to use and provides immediate value for “the network is slow” fire drills. The information gathered by AppResponse Xpert also provides input to AppMapper Xpert, to automatically draw dependency maps of critical applications.
Identifying Database Scaling Problems
A common cause of application slowness is that the application was developed with a small data set on a fast LAN development environment. Then the application is rolled out to production. It may initially run with acceptable performance. But over time, as the database grows, it becomes slower and slower. A quick analysis with AppResponse Xpert shows that one of the key middleware servers is making a lot of requests to a database server. One client request can result in many database requests or perhaps result in the transfer of a significant volume of data. Changing the database query to be more efficient typically solves the problem.
I’ve also found the case where a database server takes many seconds to return data to the middleware or application server. The application team can use AppResponse Xpert’s Database Monitoring module to identify the offending query. Sometimes a good development team can look at the user transaction and quickly determine what queries are likely to be the culprit while other times, the application is making so many database queries that a SQL query analysis tool is really what is needed. In the cases I’ve seen, the queries were poorly structured, sometimes joining large tables that resulted in extremely long query times on production data sets. Simply rewriting the queries dropped the query times by several orders of magnitude. This is where these tools pay off. The advantage using deep packet inspection on the network to identify problems with SQL queries is that there is no overhead added to the database. This is another example of how the network team can provide value to other IT teams.
Another typical example of problems within the application is the chatty conversation. One application server, or perhaps the client itself, will make many, many small requests to execute one transaction on behalf of the person running the application. It runs fine as long as the network latency between the client and server is low. However, with the advent of virtualization, the server team may have configured automatic migration of the server image to a lightly loaded host. This might move a server image to a location that puts it several milliseconds further away from other servers or from its disk storage system. A few milliseconds may not be much unless the application does hundreds or perhaps thousands of small requests to complete one transaction. Suddenly, the application goes from an acceptable level of performance to unacceptable performance. Of course, database size also affects the performance because the number of small requests goes up with the database size.
You need visibility into the number of requests between systems, where the systems are connected to the network, and the delays between requests. Getting a baseline of system performance against which you can measure future performance is extremely useful for identifying whether a given application is performing as expected and possibly identifying which server needs to be examined.
This kind of examination can be automated by AppTransaction Xpert, which can capture baseline transactions from the packet store of AppResponse Xpert and predict the change in their response times given different network parameters such as latency, bandwidth, and loss rate.
Slow Network Services
Finally, the problem may be due to slow network services. This isn’t the network itself, but services that most network-based applications depend upon for proper operation. Consider an application that makes queries to a DNS server, but the primary DNS doesn’t exist, so the app must time out the first request before attempting to query the second DNS server. I’ve seen applications that would have a 30-60 second delay upon the first execution, but would then run fine for a while. Periodically, the application would be very slow, but run fine the rest of the time. Intermittent problems are very challenging to diagnose, so this is where having something like AppResponse Xpert watching and recording all the transactions is extremely helpful. Just identify the time of the slow performance and look for something in the data. In this case, it would be an unanswered DNS request, which was successful when tried against the secondary DNS server.
Accurately diagnosing application performance can be impossible or very time consuming with the wrong tools. With the right tools and a good installation, where the tools capture the necessary data, the analysis and diagnosis can proceed very quickly. In addition, these tools not only help to defend or troubleshoot the network, but also provide value to other IT teams in the organization. I know of one site that went from not being able to help diagnose slow applications, to being able to provide deep visibility into what an application is doing from the network perspective, and providing real value to the application teams to solve the problem.
We thank Terry Slattery and OPNET for this Article