Validating Web Performance Levels with Benchmark Factory
11
pages
Slovak
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
11
pages
Slovak
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
Validating Web Performance Levels with
®Benchmark Factory
By Bernard Farrell and Kevin Dalton
Contents
Introduction ...............................................................................................................................................................3
Key Web Performance Testing Terminology .............................................................................................................3
Benchmarking...........................3
Capacity Planning .......................................................................................................................................3
Load Testing.............................4
Stress Testing...............................................................................................................................................4
Web Environments and Testing Complexities............................................................................................................4
Identifying Critical Risk Areas...................................................................................................................................4
Technologies Involved during Web Testing ...............................................................................................................5
Benchmark Factory®................................................................................................................................. ...
Validating Web Performance Levels with
Benchmark Factory
®
By Bernard Farrell and Kevin Dalton
Contents
Introduction ...............................................................................................................................................................3
Key Web Performance Testing Terminology .............................................................................................................3
Benchmarking..............................................................................................................................................3
Capacity Planning.......................................................................................................................................3
Load Testing................................................................................................................................................4
Stress Testing...............................................................................................................................................4
Web Environments and Testing Complexities............................................................................................................4
Identifying Critical Risk Areas...................................................................................................................................4
Technologies Involved during Web Testing...............................................................................................................5
Benchmark Factory®.................................................................................................................................................5
Benchmark Factory Core Components......................................................................................................................6
Visual Control Center..................................................................................................................................6
Agent............................................................................................................................................................6
Repository....................................................................................................................................................6
Benchmark Factory Modules.....................................................................................................................................8
Database Module.........................................................................................................................................9
Internet Module ...........................................................................................................................................9
Messaging Module.......................................................................................................................................9
File Server Module......................................................................................................................................9
Web Analysis and Profiling with Benchmark Factory...............................................................................................9
Real-Time Diagnostics.............................................................................................................................................10
Web Tuning..............................................................................................................................................................10
Conclusion ...............................................................................................................................................................10
About the Authors ....................................................................................................................................................11
About Quest Software ..............................................................................................................................................11
2
Validating Web Performance Levels with Benchmark Factory
®
By Bernard Farrell and Kevin Dalton
Introduction
The level of business conducted on the World Wide Web (WWW) continues to grow at
a tremendous rate. Consequently, business-to-consumer (B2C) and business-to-business
(B2B) arenas, equate to increase sales and business opportunities. Change is constant
in these arenas. Those who quickly adapt to change, profit. Those who fail to change,
struggle to stay competitive and survive. Productivity, efficiency, and customer
satisfaction require efficient Web technology. Successful e-business environments
demand Web performance tracking, validation, and evaluation.
Excellent Web performance equates to excellent customer service. Customer service
equates to maximize profits. In the highly competitive and unforgiving Internet e-
commerce environment, server performance is the defining factor that keeps customers
satisfied. Satisfied customers, in turn, provide higher company revenue and profit.
Benchmark Factory
®
is a load testing and capacity planning tool uniquely designed for
critical e-business environments. Benchmark Factory places enormous stress on a
system-under-test that typically is hard to achieve in a standard testing environment.
Systems may fail under extreme loads. Benchmark Factory identifies system capacity
and performance bottlenecks before they occur reducing downtime, development costs,
and potential loss of revenue.
Key Web Performance Testing Terminology
Web performance testing terms include:
Benchmarking
•
•
•
•
Capacity planning
Load testing
Stress testing
Benchmarking
Benchmarking is the performance test of hardware and/or software. Benchmark
Factory software can accurately test the raw power of a single machine, the interaction
in a single client/server system (one server/multiple clients), and the transactions per
second in a transaction processing system.
Capacity Planning
Capacity planning measures computer utilization of a system’s software and
applications. Capacity planners measure this computer utilization based on simulation
or mathematical models. Simulation tools like Benchmark Factory create mock
networks to gauge usage. Mathematical tools use formulas developed from previous
3
performance data to calculate expected resource consumption. Capacity planners
measure the use of host CPU, memory, and disk storage for mainframe machines. In
addition, capacity planners can measure database and network activity for server and
client machines.
Load Testing
Load testing determines system performance at maximum usage levels. The measure
of transaction times during load-testing is not as important as determining how far a
system can be pushed until it fails.
Stress Testing
Stress testing runs or stresses a program under heavy loads for sustained periods. Stress
testing determines when the performance of a system degrades over a period of time.
Web Environments and Testing Complexities
Web sites vary in complexity. Simple sites may use only static pages with a few
graphics. Sites that are more complex utilize interactive features, dynamic
(customized) content, or database connectivity. Enterprise Web traffic involves
complex interchanges between Web browsers, programming languages, operating
systems, communications protocols, servers, databases, plug-ins, security
enhancements, and firewalls. These complexities present the challenge of focusing on
what testing to perform.
In addition to Web site complexity, consider the testing environment and how it
parallels true “production” environment testing. Testing scenarios require replicated
real-life users, Internet connections, modems, communications, hardware, databases, or
other factors peculiar to the testing site. Web performance testing should strive to
duplicate a “production” system. Production differences must be considered when
developing testing plans.
Identifying Critical Risk Areas
Reviewing critical risk areas on a Web site is vital when validating Web performance
levels. Carefully consider factors that affect customer satisfaction and income
potential. The following provides a list of critical areas to review during planning
stages:
What types of problems cause the most complaints or negative publicity
•
•
•
•
•
What are the interactions between HyperText Markup Language (HTML) pages
What aspects of a site's Common Gateway Interface (CGI), applets, ActiveX
components, database interfaces, Active Server Page (ASP), and dynamic page
generators, are most complex
What Web site functionality is most critical to its purpose
What site areas require the heaviest database interaction
4
What site areas are the most popular and generate the most traffic
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
What are the TCP/IP communications/firewalls requirements
There are a variety of servers and browsers used for Web e-commerce activity.
Numerous versions of servers and browsers, variations in connection speeds, rapidly
changing technologies, and multiple standards and protocols create testing challenges,
and force major, ongoing Web site testing.
Technologies Involved during Web Testing
Carefully consider the technologies used on a Web site. Reviewing and understanding
the interaction of applicable site technologies provides the basis for obtaining accurate
testing results. Benchmark Factory supports the following technologies:
Static HTML
Dynamic HTML
Multimedia
Common Gateway Interface (CGI)
Application Server Page (ASP)
JavaServer Page (JSP)
Hypertext Preprocessor (PHP)
JavaScript
Perl
VB Script
Java DataBase Connectivity (JDBC)
Java
Plug-ins
Structured Query Language (SQL)
Proxies
Secure Sockets Layer (SSL)
Web servers
Application servers
Cache servers
Load balancers
Benchmark Factory
Benchmark Factory provides all of the essential functions necessary to test an e-
commerce application by developing custom server benchmarks. Simulated throughput
provides an overall measure of server performance, including server software and
hardware performance; throughput performance; and total system throughput.
The operations contained within Benchmark Factory represent real applications doing
real work. They contain specially written transactions (tests) that mimic real-world
5
client/server application transactions, without actually running the client side
application.
Benchmark Factory provides an accurate method of collecting and measuring
transactions per second (TPS) and other system-under-test statistics. Benchmark
Factory allows a client/server developer to create custom benchmarks. Properly
developed custom benchmarks, allow testers to draw conclusions about server
performance, transaction design, and overall solution stability. Benchmark Factory
supports a variety of server systems, including client/server database servers, file
servers, messaging servers, transaction servers, cache servers, and Internet servers.
Benchmark Factory Core Components
Benchmark Factory consists of the following core components:
Visual Control Center
•
•
•
Agent
Repository
These core components are multi-threaded 32-bit Windows applications capable of
running on Windows NT and Windows 2000 platforms.
Visual Control Center
The Visual Control Center is the dashboard of the Benchmark Factory testing
environment. It is where test scenarios run and are coordinated. The Visual Control
Center monitors test progress, collects statistics, and performs data analysis.
Agent
Agents are workstations that run the Agent software. The Agent software sends
requests to a system-under-test. Agents are multi-threaded, and simulate more than a
thousand users on a single computer. Benchmark Factory is infinitely scaleable. It
scales horizontally (supports hundreds of workstations running the Agent software),
and vertically (each Agent is capable of running a thousand threads.)
Repository
The repository stores database results. The Visual Control Center inserts testing results
into the repository and provides access to its data. In addition, the repository houses all
test scenarios with settings and options. By default, the repository is a Sybase SQL
Anywhere database that resides on the same machine as the Visual Control Center. If
required, the repository can be changed to a Microsoft SQL Server database.
6
Figure 1 provides an illustration of Benchmark Factory components.
Figure 1—Benchmark Factory Core Components
Benchmark Factory collects a vast amount of statistics, such as overall server
throughput (measured in transactions per second/minute, or bytes transferred), and
detailed transactions executed statistics by individual workstation producing a testing
load. These statistics measure, analyze, and predict the capacity of a server system.
Benchmark Factory software is a stand-alone benchmarking program that loads on a
server or workstation and does not require third party or custom applications.
7
Benchmark Factory Modules
Benchmark Factory provides the versatility to benchmark all testing platforms
adequately. Companies depend on the Web for revenue. Web site downtime quickly
leads to a loss of revenue, poor user perception and loss of customer loyalty.
Companies can determine future hardware and software investments to meet expected
e-commerce demands by performing adequate capacity planning.
Quest Software meets the wide range of Web performance testing requirements by
packaging Benchmark Factory as a complete suite of testing platforms, or by allowing
the purchase of individual platforms (Figure 2.) Benchmark Factory supports the
following testing platforms:
Database
•
•
•
•
Internet
Messaging
File Server
Figure 2—Benchmark Factory Modules
Unlike other testing tools, Benchmark Factory creates realistic tests without
programming and provides comprehensive, easy-to-understand reports.
8
Database Module
The Benchmark Factory Database module allows IT managers and database
administrators to ensure their database applications achieve the required service levels
of performance and availability by determining true database performance, and
identifying the point-of-system failure.
Internet Module
The Benchmark Factory Internet module allows Web site administrators to ensure their
Web applications achieve required service levels of performance, availability, and
identify the point-of-system failure.
Messaging Module
The Benchmark Factory Messaging module allows messaging administrators to ensure their
email servers achieve the required service levels of performance, availability, and identifies the
point-of-system failure.
File Server Module
The Benchmark Factory Internet module allows IT Managers to ensure their file servers
achieve the required service levels of performance and identifies the point-of-system
failure.
Web Analysis and Profiling with Benchmark Factory
Benchmark Factory records essential performance metrics during each test.
Administrators use these metrics to determine the most effective course of action to
improve performance. Benchmark Factory illuminates the behavior of a system-under-
test to completely review a system’s architecture.
Benchmark Factory provides modeling and reporting abilities, making it an invaluable
tool for measuring server performance. Creating realistic users and executing
transactions against a server becomes a trivial task. The addition of script functions
expands a tester’s ability to simulate further realistic user models. Modeling realistic
users and measuring the transaction times and throughput, at different user loads,
presents a clear picture of system capacity. Benchmark Factory gathers numerous
metrics that include:
Response time
•
•
•
•
•
Retrieval time
Transaction time
Transactions per second
Bytes per second
9
Real-Time Diagnostics
Benchmark Factory produces real-time graphs and raw data when running a test.
During initial benchmarking, real-time analysis provides adequate information to
perform minor tuning and initial capacity estimates. These real-time graphs and raw
data may be sufficient to identify obvious transaction problems, or estimate maximum
user load.
Web Tuning
The process of using Benchmark Factory to tune a Web server involves the following
steps:
1. Running a test against a Web server
2. Reviewing the recorded metrics and other performance factors
3. Tuning a server to improve performance
This test cycle is summarized as; test, tune, and repeat. The following provides real-
world testing examples:
An administrator notices a particular Web server has performance problems and feels it
is memory related. An administrator runs a test against a server. After completing a
test, an administrator increases the physical memory in a machine. Next, an
administrator runs the same test again and compares the test results
•
•
•
•
•
An administrator has an interactive Web site with a database used to put dynamic
information into the Web pages. The database resides on the same system as the Web
server. An administrator feels that the database could scale to higher user loads if the
processes were on separate servers. To test this theory, an administrator tests a server
at different user loads, then splits the database off to another server and re-runs a test
using the same user loads. An administrator then compares the results to determine
where the two architectures diverge in performance
Conclusion
Use the best tools available when benchmarking e-commerce sites. An effective e-
commerce site must scale with the number of users, respond reliably and quickly, and
optimizes its use of bandwidth. Benchmark Factory provides all of the essential
functions necessary to test e-commerce applications. In addition to Benchmark
Factory, Quest Software offers the following tools to monitor and tune e-commerce
systems during capacity planning, load-testing, and normal operation:
Spotlight
®
—Provides essential functions to tune specific systems. Quest Software
offers Spotlight for Oracle, SQL servers, and Web servers
I/Watch™
—Provides essential functions to monitor and tune Oracle and SQL Server
databases with trend analysis and drill-down capabilities
Foglight®
—Provides complete enterprise monitoring functions for Web servers and
database servers
10
FunnelWeb Suite®
—Provides a complete solution set to manage a Web site
•
About the Authors
Bernard Farrell is a seasoned technical writer for Quest Software. Mr. Farrell currently
writes technical documentation for Benchmark Factory. Mr. Farrell's writing
experience centers generally on benchmarking, and specifically on database
applications.
Kevin Dalton, P.E., currently is the Development Manager of Benchmark Factory for
Quest Software. Mr. Dalton's responsibilities focus on the day-to-day development of
the Benchmark Factory application. Mr. Dalton was the initial architect/developer of
the Benchmark Factory Web Module and brings an excess of 12 years of technical
expertise of which five of those years centered on performance and load testing. Prior
to joining the Quest team, Mr. Dalton was a consultant for the Anheuser-Busch
companies.
About Quest Software
Quest Software, Inc. (NASDAQ: QSFT) is the leading provider of application
management solutions. Our software gives businesses confidence that their vital
applications will be available and performing well — while simultaneously driving
down the costs of managing them. By focusing on the people and technology that
make applications run, Quest enables IT professionals to achieve more with fewer
resources and get the most out of existing application investments. Founded in 1987
and based in Irvine, California, Quest Software has offices around the globe and more
than 1,800 employees. Approximately 10,000 businesses worldwide use Quest
Software because
business runs better on Quest
.
For more information on Quest and Benchmark Factory, visit
www.quest.com
.
11
