Application Server Benchmarking Fundamentals
9
pages
Slovak
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
9
pages
Slovak
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
Application Server Benchmarking Fundamentals
By Prashanth Ponnachath, eBD SWAT Team, Sybase Inc
1 Table of Contents
1 Table of Contents ................................................................................................................................... 2
2 Executive Summary.................. 3
3 Definitions.............................................................................................................................................. 3
3.1 Benchmark...................................................................................................................................... 3
3.2 Application Server.......................................................................................................................... 3
3.3 Web Server ..................................................................................................................................... 3
4 Benchmarking Strategy .......................................................................................................... 3
5 Stress test vs. Performance test............................................................................................................... 4
5.1 Examples of flawed application server stress tests......................................................................... 4
6 A sound application server benchmark.......................... ...
By Prashanth Ponnachath, eBD SWAT Team, Sybase Inc
Application Server Benchmarking Fundamentals
1 Table of Contents
1
Table of Contents ................................................................................................................................... 2
2
Executive Summary................................................................................................................................ 3
3
Definitions.............................................................................................................................................. 3
3.1
Benchmark...................................................................................................................................... 3
3.2
Application Server.......................................................................................................................... 3
3.3
Web Server..................................................................................................................................... 3
4
Benchmarking Strategy .......................................................................................................................... 3
5
Stress test vs. Performance test............................................................................................................... 4
5.1
Examples of flawed application server stress tests......................................................................... 4
6
A sound application server benchmark................................................................................................... 5
6.1.1
The bookstore home page....................................................................................................... 6
6.1.2
The browse screen .................................................................................................................. 7
6.2
The search screen............................................................................................................................ 7
6.3
The testing process ......................................................................................................................... 8
7
Conclusion.............................................................................................................................................. 9
2
2 Executive Summary
The goal of this paper is to point out fundamentals involved in an application server
benchmark. The paper also points out some of the common inaccurate and incorrect
benchmarking practices followed in the industry. Finally, an example of a good
benchmark is illustrated and the metrics explained.
3 Definitions
3.1 Benchmark
A benchmark is a process by which two entities that perform or claim to perform similar
functions are compared. It is a process by which the core functionalities of products are
tested and assessed.
3.2 Application Server
An Application Server is a software container that provides middleware services to server
side components (like Enterprise Java Beans, .Net, Power Builder, CORBA) to manage
them. It provides a run time environment for the deployed components. It is often used
for complex transaction based applications.
3.3 Web Server
A Web Server is a program that delivers web pages or html pages to clients that request
them. The clients request specific pages by contacting the web server on a specific
address called the IP address using the http protocol. On receiving the request, the web
server delivers the page to the client. Some web servers can handle dynamic web content
like JSP, servlets, ASP, Power Dynamo.
4 Benchmarking Strategy
The common sense approach followed in any successful benchmark would involve the
following fundamental steps.
•
Understanding the core functionalities of the products being benchmarked.
•
Creating tests that measure the core functionality and, simulate real world
environments.
•
Conducting the tests in a stable environment i.e. all the products should be tested
in the same environment.
•
Assessing and measuring the core functionalities the products.
•
Reporting results in a clear and concise format.
Based on knowledge of the core functionalities of an application server, the following
functionalities need to be measured and assessed during an application server benchmark.
•
High availability:
How many users can be supported? In an enterprise
environment an application server should be able to scale up to several
simultaneous users performing a variety of transactions.
3
•
Ability to handle complex transactions using EJBs:
The application server
must be able to support entity, session and message driven beans ( in case the
application follows the J2EE 1.3 spec which in turn supports the EJB 2.0 spec).
•
Response times to handle the complex transactions:
In addition to supporting
several simultaneous users, the application server must be able to do so in an
efficient manner i.e. the users must served with good response times.
•
Support for complex databases:
The backbone of an enterprise application is
the database. Hence an application server must able to support the complex
database transactions an enterprise application entails.
5 Stress test vs. Performance test
When performing a benchmark, it is very essential to understand the difference between a
stress test and performance test. A stress test measures the maximum number of users an
application server can handle before it fails. For example, a bookstore application may be
able to support only 2000 users for a given hardware and software environment, whereas
a performance test measures specific metrics like response time and throughput for a
finite number of users for a given software and hardware environment.
5.1 Examples of flawed application server stress tests
Case-1
: Testing dynamic content access without component access
This is one of the most common mistakes made while testing an application server. Here,
only the web server and the servlet engine of the application server are being tested. In
other words, the test application typically contains JSPs without any server side
components like EJBs. This kind of testing is analogous to benchmarking cars by only
testing the battery life and ignoring other core features like engine performance,
transmission etc. Moreover, if this test is simulating a realistic scenario then an
application server investment in this case is a waste. A web server with a servlet engine
would serve the purpose.
4
Case-2
: Testing a static html page
Here, the application server is tested by sending http requests to the web server that is
bundled with it. This test case a worse than the previous case because the only
functionality being tested here is the performance of the web server. It is analogous to
benchmarking cars by testing the stereo systems in them and not bothering to test core
functionalities like the engine performance, transmission etc.
6 A sound application server benchmark
In a sound application server benchmark, the core functionalities of the application server
are tested.
As shown in the figure, the test application must contain server side components like
EJBs in the benchmark. A good application server benchmarking application must
be a full-fledged e-business application that includes EJBs, servlets and JSPs. Moreover
the test application must:
5
•
Work on all the application servers and on all versions
•
Simulate a real world environment
•
Test all EJBs , servlets and JSPs deployed
•
Perform complex database transactions in a highly populated database
•
Test the object caching, JSP caching and connection pooling functionalities of an
app server
An example of a sound application benchmark is the Nile.com benchmark that was
developed by Doculabs (an independent consulting firm). This benchmarking application
is a real world example of a bookstore called Nile.com (like amazon.com) where users
can browse books, search books, and buy books online. This application is written using
Enterprise Java Beans, JSP, and servlets. The EJBs access a database populated with a
million rows of customers and books. This benchmarking application coupled with a
good load-testing tool make an excellent combination to perform an application server
benchmark. The screen shots of the Nile.com bookstore application are shown below.
6.1.1 The bookstore home page
This is the home page of the Nile.com application. A user can use the navigation bar on
the left to browse and search books. The user can also view his shopping cart by
navigating through the link.
6
6.1.2 The browse screen
A user can view the list of subjects in this page and can browse books based on the
subjects.
6.2
The search screen
7
The search utility in the Nile.com application enables a user to search the million books
in the database based on title, author name and subject.
6.3 The testing process
Once this application is deployed in an application server, any load testing tool like e-test
or openSTA can be used to simulate users to perform stress tests or performance tests. A
stress test can also be performed by using a load testing tool like silk performer. The
metrics from a test is either generated by the testing software or can be manually
calculated. In a stress test the metric would be the maximum number of users supported
by the application server for the given hardware and software environment. In a
performance test, metrics like
throughpu
t (number of tasks/second)
and http
responses/second
can be ascertained and the values can be plotted. An example of such a
plot is shown below. A Nile.com performance test conducted in our lab to benchmark
EAServer and three other leading application servers yielded the following results as
shown in the graphs. The testing software used was openSTA
(
http://www.openSTA.org
).
6.3.1.1
The throughput comparison
The graph shows the throughput of EAServer vs. the throughput of three other
application servers. Throughput is the number of completed tasks (like browsing,
searching etc) per second. Higher throughput signifies better performance since more
tasks are being performed in less time. The graph is a plot of the number of virtual users
on the x-axis vs. the throughput on the y-axis. The above graph indicates that EAServer
not only produces better throughput to start with but it also scales well as the load
increases.
8
6.3.1.2
The http responses/second comparison
The graph above is a plot of http responses vs. number of virtual users. The higher the
number of the http responses per second, the better the performance. In the graph above
EAServer produces higher http responses per second compared to the other application
servers. This also indicates that the response time for a task is faster for EAServer.
7 Conclusion
A good application server benchmark is one that simulates a real world enterprise
scenario and tests the core aspects of the server functionality. Moreover the results and
metrics should be presented in a clear and concise format.
9
