Time is Money. Understanding application responsiveness and latency is critical not only for delivering good application behavior. It is critical for maintaining profitability and containing risk. But good characterization of bad data is useless. When measurements of response time present false or misleading latency information, even the best analysis can lead to wrong operational decisions and poor application experience.
This presentation discusses some common pitfalls encountered in measuring and characterizing latency. It demonstrates and discusses some false assumptions and measurement techniques that lead to dramatically incorrect reporting results, and covers simple ways to sanity check and correct these situations. It discusses the fallacy of using standard deviation measurements, the strongly multi-modal nature of latency, common discontinuities found in most computing platforms, and how back pressure and coordinated data omission issues can literally skew measurement results by orders of magnitude. This presentation introduces and demonstrates how simple and recently open sourced tools can be used to improve and gain higher confidence in both latency measurement and reporting.