The Bottom Line of Benchmarking Basics
15
pages
Slovak
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
15
pages
Slovak
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
This is a blank for complete User Guide. Contains all required sections and boilerplates.
Voir
®
The Bottom Line of Benchmarking Basics
White Paper
written by Bernard Farrell Applications and Database Management Quest Software, Inc.
© Copyright Quest® Software, Inc. 2005. All rights reserved.
The information in this publication is furnished for information use only, does not constitute a commitment from Quest Software Inc. of any features or functions discussed and is subject to change without notice. Quest Software, Inc. assumes no responsibility or liability for any errors or inaccuracies that may appear in this publication.
Last revised: April 2005
T ABLE OF C ONTENTS
INTRODUCTION ........................................................................................................................... 4 THE BENCHMARKING PROCESS.............................................................................................. 5 WHY BENCHMARK A DATABASE?........................................................................................... 6 BASIC BENCHMARK COMPONENTS ........................................................................................ 7 S PECIFICATIONS ......................................................................................................................... 7 Details....................................................................................................................................7 Design Goals ......................................................................................................................... 7 Data Points ............................................................................................................................ 8 Execution Plan.......................................................................................................................8 C ONTROL L OGIC ......................................................................................................................... 8 Repeatability..........................................................................................................................8 Accurate Statistics ................................................................................................................. 8 IMPLEMENTING THE BENCHMARKING PROCESS WITH THE RIGHT TOOLS ..................... 9 B ENCHMARK F ACTORY C OMPONENTS .......................................................................................... 9 B ENCHMARK F ACTORY C ONSOLE .............................................................................................. 10 B ENCHMARK F ACTORY A GENTS ................................................................................................. 11 R EPOSITORY ............................................................................................................................. 11 B ENCHMARK F ACTORY W IZARDS ............................................................................................... 12 CONCLUSION ............................................................................................................................ 13 ABOUT THE AUTHOR ............................................................................................................... 14 ABOUT QUEST SOFTWARE, INC. ........................................................................................... 15 C ONTACTING Q UEST S OFTWARE ............................................................................................... 15 T RADEMARKS ............................................................................................................................ 15
The Bottom Line of Benchmarking Basics - Bernard Farrell, Quest Software, Inc.
3
I NTRODUCTION
Margins of profit in a lightening quick e-commerce world are thin enough without problems like haywire databases that result in disappointed customers and lost revenue. How do IT managers stay ahead of their databases and ensure reliability 24/7? They do it by benchmarking their database and understanding that a key factor in the benchmarking process centers on giving developers and DBAs the right tools.
The Bottom Line of Benchmarking Basics - Bernard Farrell, Quest Software, Inc.
4
T HE B ENCHMARKING P ROCESS
Benchmarking is a performance test of hardware and/or software. Benchmarks measure the:
• Capacity of a system (often referred to as capacity planning) • Performance of a particular application, such as a database
All good benchmarks use a well-defined testing methodology based on real-world use of a computer system. Benchmarks measure system performance in an accurate and repeatable manner, allowing IT professionals to properly judge the performance and capacity of a system-under-test.
The Bottom Line of Benchmarking Basics - Bernard Farrell, Quest Software, Inc.
5
W HY B ENCHMARK A D ATABASE ?
Reliable database performance is essential for excellent customer service. Excellent customer service and satisfied customers mean maximized profits. In a highly competitive and unforgiving e-commerce environment, database performance is the defining factor that keeps customers satisfied. Implementing a well-planned benchmarking program is essential to staying competitive, and the entire process begins with benchmarking basics.
The Bottom Line of Benchmarking Basics - Bernard Farrell, Quest Software, Inc.
6
B ASIC B ENCHMARK C OMPONENTS
Benchmarks are made up of the following basic components:
• Specifications • Control logic • Implementation
Specifications
Benchmark specification documentation provides the following information:
Details • • Design goals • Data points • Execution plan
Details Benchmarking details define metric objectives. The Transaction Processing Counsel (TPC) is a non-profit corporation that defines transaction processing and database benchmarks. The TPC disseminates objective, verifiable performance specifications to the IT industry. For example, the TPC-C specification defines online transaction processing benchmarks.
Design Goals Well-defined design goals are essential for benchmark success. A benchmark must possess the following characteristics:
• Scalability Benchmarks, like enterprise systems, must be able to serve large numbers of users in a testing process without requiring major changes. • Easy-to-Use and Understand DBAs and others involved in the benchmarking process are busy people. They do not have extra hours to analyze complicated benchmarks. A benchmarking system should be easy to install, run and understand.
The Bottom Line of Benchmarking Basics - Bernard Farrell, Quest Software, Inc.
7
• Representative of the Tested Workload The workload being tested must accurately represent a system-under-test. For example, if a system-under-test is a database, the workload must reflect valid test data measurements, improved customer database access time and other factors important to the overall operation of a database. • Accuracy Benchmark testing must accurately reflect the demands placed on enterprise systems. During the day-to-day operations of a database system, peak demands can quickly overload a system, causing customer dissatisfaction and profit loss. Benchmark testing accurately defines those demands avoiding profit-robbing system overloads.
Data Points Data points are the result of a benchmarking test. They represent the summary of the specified results for a userload. Data points provide the detailed information required to make accurate decisions on the performance capability of a system-under- test. An example of a data point is average transactions per second (TPS) at 200 users. In addition, data points must be defined before implementing the benchmark testing process.
Execution Plan All testing processes require a well-designed execution plan. This ensures real-world environments are duplicated during a benchmarking process.
Control Logic Control logic in the benchmarking process must have a well-defined initialization sequence that is separate from test and data collection. The control logic is not part of the benchmark. However, a benchmark does depend on the control logic for the following:
• Repeatability • Accurate statistics
Repeatability Repeatability represents benchmark results from multiple runs of the same test, with consistent configuration test environments and parameters. If a benchmark is not sufficiently repeatable, the results have no meaning.
Accurate Statistics Accurate statistics are essential during a benchmarking process. Test result data, for example, Transactions Per Second (TPS), provides a user with the data necessary to determine where bottlenecks or potential enterprise problems may occur.
The Bottom Line of Benchmarking Basics - Bernard Farrell, Quest Software, Inc. 8
I MPLEMENTING THE B ENCHMARKING P ROCESS WITH THE R IGHT T OOLS
Benchmark Factory ® for Databases allows database professionals to implement benchmark specifications and control logic. Benchmark Factory is an easy-to-use load-testing and capacity-planning tool for critical database environments. It places enormous stress (that, typically, is hard to achieve) on a database in a standard testing environment. A database usually breaks under extreme load when it is needed the most. By identifying system capacity and performance bottlenecks before they occur, Benchmark Factory facilitates proactive testing that reduces downtime, development costs and potential loss of revenue. Benchmark Factory provides the ability to:
• Simulate thousands of concurrent users with a minimal amount of hardware, providing scalability to simulate large numbers of users. • Control agents installed on client machine to spawn multiple virtual-user sessions, allowing the workload to be representative of the database being tested. • Determine database throughput and capacity with reliable data points. • View real-time testing results providing immediate feedback on database performance allowing an accurate assessment of a critical database system. • Examine data points with a built-in report viewer or printed reports.
Benchmark Factory Components Benchmark Factory provides the following components:
• Benchmark Factory Console • Benchmark Factory Agents • Repository
The Bottom Line of Benchmarking Basics - Bernard Farrell, Quest Software, Inc.
9
Benchmark Factory Console
The Benchmark Factory Console (Figure 1) is the graphical user interface used to implement the load testing process and controls one or more distributed Agents. Workloads are created in the Benchmark Factory Console, allowing the user to set the desired scalability and control logic for the database being tested. Workloads can be created with the desired number of users, submitted to test and then repeated to gather accurate benchmarking statistics.
Figure 1-Benchmark Factory Console
The Bottom Line of Benchmarking Basics - Bernard Farrell, Quest Software, Inc.
10
Benchmark Factory Agents
Benchmark Factory Agents (Figure 2) simulate virtual users. This allows DBAs and administrators to build workloads that are representative of the database being tested. Agents are scalable and can simulate thousands of users, requiring no major changes to the testing process. Each simulated virtual user executes transactions and records statistics. The Agents send transactions to the database under test. The Agents also record statistics that include how much data the resulting transaction contained, and how long it took to get the results. At the end of a testing iteration, each Agent reports accurate data points back to the Benchmark Factory Console.
Figure 2-Benchmark Factory Agent
Repository
The Repository is a database where all testing results are stored. The Benchmark Factory Console inserts test results into the repository and provides an easy way to access data. By default, the Repository is a Sybase SQL Anywhere database that resides on the same machine as the Benchmark Factory Console. The Repository can be changed to a Microsoft SQL server database if required.
The Bottom Line of Benchmarking Basics - Bernard Farrell, Quest Software, Inc.
11
