We have worked with JMeter preparing scripts to simulate user actions. Now, it’s time to prepare test executions in BlazeMeter to simulate larger scenarios. It’s very important that all JMeter concepts are clear. Let’s review the most important JMeter elements, focusing on the scope and structure.
In our scripts we want something like the structure shown in the following image (taken from here):
In this script each request has its own HTTP Header Manager and Response Assertion, applied only to the corresponding request. In some cases, requests can have a Regular Expression Extractor.
The JMeter test tree contains elements that are both hierarchical and ordered. Some elements are strictly hierarchical and some are primarily ordered.
In this test plan, we created a list of sample HTTP Requests. These requests are organized within Transaction Controllers. Both HTTP Requests and Transaction Controllers are ordered, therefore the order of the requests will be, 1, 2, 3, 4, 5 and 6.
Elements like Response Assertion, Regular Expression Extractors and Timers are hierarchical. So, after JMeter executes the HTTP Request 1, it will execute the Response Assertion 1 and the Regular Expression Extractor 1. This behavior is the same for all HTTP Requests.
If we have a test plan like this:
Then Assertion #1 is applied to Request One and Assertion #2 is applied to Request Two and Three.
If a hierarchical element is applied to a controller, then this element will be applied to all the elements inside said controller. Timers are inside a Simple Controller so that the timer executes only once. These Simple Controllers are between the Transaction Controllers because they represent the time that users spend thinking or filling out the forms on the page and each Transaction Controller contains all the HTTP Request sends in each action.
Timers are also hierarchical and if we do something like in the following example, then Timer #1 will apply to Request Two, Threeand Four and Timer #2 will apply to all Requests.
About BlazeMeter
JMeter is one of the best open source performance testing tools available on the market today, but no tool is foolproof and there are always pros and cons to be found with each one.
If we want to excecute a performance test using JMeter, depending on the script, we can probably execute around 500 users with the load generator. In our experience, sometimes we can execute a few more users and other times we can only execute less than 100.
With JMeter we can distribute the load between various generators. In this way, if we wanted to simulate 1,000 users then we could use two generators, each one executing 500 users.
This is a good solution if we have a small load to generate, but if we wanted to generate 10,000 users then we would have to use 20 generators, which would be too expensive and complex.
What is BlazeMeter?
BlazeMeter is “JMeter in the cloud”. This means it’s not only 100% compatible with JMeter, but it also addresses it’s limitations like scalability, stability and reporting.
How does it work?
In BlazeMeter we can directly upload a JMeter script and execute it.
All we need to do is write the test-script or upload JMeter scripts, choose the amount of load-engines and run the test. BlazeMeter takes care of the everything else. An unlimited number of load-engines are pre-configured and available. Detailed graphical reports are generated during the load as well.
Benefits
- The ability to generate up to 1,000,000 (or even more!) virtual users — no need to worry about infrastructure cost and setup
- The ability to test from multiple geo-locations — no need to worry about test and report synchronization
- Real time monitoring and reporting
- Easy access to monitoring and reporting
- Easy access to historical reports for comparison
- The option to extend your test data further with top-tier APM solution integrations like New Relic and CloudWatch
- End-to-end visibility of your server, app (web and mobile), and end user experience
- “On-the-fly” script recording with the BlazeMeter Chrome Extension
- It’s a simple way to maintain and execute JMeter scripts from one location
- Real mobile device testing
For more information visit the official BlazeMeter webpage.
JPETER
Introduction to Regular Expressions
Introduction to Regular Expressions
A regular expression is a tool we can use to search for and manipulate text, based on patterns. They come in very handy for test automation, mainly when we need to extract certain data from an HTML, in order to load that into a variable, and use the variable to send the extracted data in a following request or requests. You will see that you will need to get familiar with the use of regular expressions in order to automate dynamic websites. A standard usage example is to get any ID from the server response that must be passed to the following requests in order to identify the corresponding element identified by the ID (which varies for each user or each execution or for each test data).
Regexes, short for regular expressions, are used in many contexts, even in text editors, when you want to search or replace a string by other. For example, if I want to find all the email addresses in a document, I could search all the string pieces containing some text, an @, some other text, dot and a couple of characters, and optionally, a dot and a short sequence of characters. There is a language to represent the patterns (actually, there are different implementations, so, there are different languages, but all are similar, based on the same premises).
Suppose we want to match the following portion of a webpage to extract an ID:
name=”productID” value=”123abc” >
A suitable regular expression would be:
name=”productID” value=”(.+?)” >
Where as you can see there are special characters:
- ( and ) : enclose the string that matches and is returned.
- . : match any character.
- + : one or more times.
- ? : stop at the first match.
If you want (or need) to learn more about regexes, refer to http://www.regular-expressions.info/, but our favorite is this one: http://regexone.com/ because you can learn in a very practical way. Give it a try!
Also, there is a tool to test your regexes http://www.regextester.com.
Be careful, in any of these tools there could be some differences between the webpage and the JMeter regex execution engine.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
Data parameterization
Well, we already are able to prepare a script that can work properly, simulating user actions, right? Following what we have just seen, we can simulate any kind of sequence of user actions. Do you agree?
Actually, this is not true. The truth is that there are certain kinds of things for which we need to do something else, and here we will understand what.
Imagine that we already prepared a script that perfectly simulates a user accessing the OpenCart site, adding a product to the cart, and paying for it. After the variable correlation, with the cookie management done, and all the required validations in their place, we start to run some tests. Think that after each run we are decreasing the stock for this product and there will be a moment when this product will not be available any longer. The problem here is that we are reproducing the same sequence of actions with the same data, the data that we used when we recorded the script.
Believe it or not, this kind of situation can be even worse. Imagine now that we start to run concurrent virtual users. If we recorded the script with the user “Lucia” and reproduced the script with 500 virtual users, there would be 500 virtual users trying to use the same username and password. In many sites that wouldn’t work because they don’t allow more than a certain amount of simultaneous sessions. Anyway, even if the system supports this kind of situation, is it the same for a system to respond to 500 different users than responding to 500 users accessing the same account? The answer is NO because in the second option, there are aspects that could positively affect the system and some other aspects that could have a negative impact. For example, if all the users accessed with the same username, the data extracted from the database would be cached after the first user accessed, and then, the rest of the users would not access the database, but the cache (which is much faster). On the other hand, if all the users accessed with the same key a certain operation, for example, to buy the same product, they would be locking the same register in the database, causing more concurrency locks than what would normally happen in production.
That’s why it makes sense to talk about parametrization. Basically, it consists of varying certain data in the scripts, taking their values from an external source, i.e., a CSV or XML file. In that way we can simulate 500 users with different usernames and passwords, each of them buying different products.
We will see in following videos how to do that in JMeter, but from a methodological point of view, I’d like to point out that it’s very important how we chose which variables to parametrize and which data we use. I’d say that it’s very important to pay attention to a real database from the production environment and analyze which data we should use. Imagine that we have automated a script for a user accessing the system then checking his order history. It’s not going to be the same if the user has no records in this query than if it’s a regular buyer who has lots of entries for this menu. The latter will be the most common situation and the most stressful for the server.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
Introduction to Controllers
Introduction to Controllers
We have the possibility of giving structure to our scripts, making them easier to read, understand and maintain. That is: making them with quality. Sometimes we are in need of adding certain logic to our scripts, for example, using structures as loops, if-then-else, randomize some requests, etc. All these flow management aspects are covered in JMeter with “Controllers”. We could say that Logic Controllers determine the order in which the requests are executed.
Some controllers are very useful for structure; they do not provide a special sequence dictating how the requests are going to be executed. For instance, Simple Controller, or Recording Controller are useful to hold groups of HTTP requests. We tend to prefer the “Transaction Controller” because it adds a very useful functionality: when you execute the load, you will have not only the information about the response times for each HTTP request, but also for each Transaction Controller, which will be the sum of all the requests contained in it. That’s useful to report how much time it takes to load the page. You have to pay attention to all the requests involved in this action. The same goes for when we are accessing any other functionality, we can group all the related requests and have a summary with the response times for the whole group of actions, not only for each HTTP requests separately.
There are other useful controllers, such as the Once-Only-Controller. A typical situation when it’s needed is when you want to execute the login section only once, at the beginning, but then it is not necessary any longer. In this case, you put all the requests related to the login section inside a Once-Only-Controller, so that this section will only be executed once. Also, the Include Controller is used to include a module from a different Test Plan into the script.
Here you have the full documentation about this topic in the JMeter user manual. In the following videos we will pay attention to the most important and most used controllers.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
Test scenarios
What does a test scenario mean?
Ok, we are almost done:
- We recorded some scripts, simulating the main test cases (signing up in the site, searching products, buying items, etc.)
- We prepared all these scripts properly: we added validations, we correlated variables, we parameterized some data, prepared the logic flow, etc.
Now, we are approaching the most entertaining moment for a performance tester: the load test execution! We want to see the CPU increase up to the maximum capacity, we want to look for bottlenecks (and also be able to find them!), we want to improve the system and contribute to the project. If you think about it, until now, our organization hasn’t benefited from our work and it has been hard work preparing the whole load test simulation.
What else do we need in order to be able to execute our load simulation? Well, that would be something that is known as a “test scenario”.
Test Scenario
Basically, test scenario refers to how the different users in the whole load behave, how many of them execute each test case, each script, how many times and for how long. We can say that we will execute a script mix, and we will do that using Logic Controllers and Thread Groups. But, before we continue with this, we need to understand something that we call “specialized users”.
Specialized Users
Our virtual users are specialized, which means that what each virtual user actually will do is repeat the same action over and over again. For example, a virtual user will access the site with their credentials (only once) and then will enter a loop doing the same group of actions until the end of the test, e.g., searching for different products. Then, we will have another group of users specialized in something else, instead of repeating the product search, they will sign up with different usernames and user data. But, this is not real; this is not simulating accurately what will happen in the real world in production. Right?
Equivalent Load
Even though this is not exactly what happens in production, for the server it will be an equivalent load, and it will use an equivalent effort to respond to real users and to respond to specialized users. We have to prepare our scripts properly in order not to let the server realize the difference between real users and virtual users, which implies carefully designing the script mix, the test scenario.
Imagine that after an analysis, the experts arrived to this simple mix of cases, as the expected load in its peak hour in production:
- 100 users following these steps:
- Sign into the system.
- Search 5 different products.
- Check out the cart once.
So, we could say that in total the server received:
- 100 attempts to login.
- 500 product searches (5 from each of the 100 users).
- 100 payments.
We could design an equivalent scenario with specialized users as represented in the following table:
Number of concurrent usersNumber of repetitionsTest case 01 — sign in and search5010Test case 02 — sign in and pay 502
If we consider that the login section is put inside a Once-Only-Controller, then we can arrive at the conclusion that the load is equivalent, it will execute 100 logins, 500 product searches (10 from each of the 50 users in Test case 01) and 100 payments (2 from each of the 50 users in Test case 02).
When designing the test scenarios, we should arrive at something like the table shown above because it has some advantages:
- We can measure separately the response times for each test case.
- It’s easier and clearer to implement in any load simulation tool.
- It’s easier to analyze the statistics of a site in production and arrive to a design like this.
- It’s easy to read, understand and modify.
- It’s clear how to scale, how to execute a load test doubling the stress on the server.
Advanced aspects of Test Scenarios
There are a couple of very important concepts to look at before we start putting it all together and into practice in JMeter.
Ramp-up
If you prepare a load simulation where a whole bunch of users starts when you press the play button, it’s like coordinating a group of people so that they all click at the very same moment. That will never happen as it’s not a common situation. The worst part: it’s more stressful for the server than what actually happens in production, which means that we will deliver very negative results to our team, when perhaps there are no such problems.
That’s why it is very important to design the load with a proper ramp-up, that is: a period of time at the beginning of the test where the amount of active users increases progressively. This strategy avoids the problem of shocking the system by going from no load to a very heavy load from one second to the next. The difference in the ramp-up and the stress it would give is like waking someone up gently versus throwing ice cubes all over their bed! Don’t be that rude to the servers!
Test duration
You have to define for how much time you want your test to be running. It could be the time it takes to finish all the actions you planned in order to measure how much time it takes to process all the data.
The strategy we typically use is define one-hour tests. We take this into account from the beginning, when planning and designing the test scenarios. It’s easier to think about “the peak hour” and try to simulate this hour of intense work. Also, we have seen that it’s a prudent time slot to have meaningful results without waiting for so long to get something to show.
Depending on the context, you would define your tests with more or less than an hour, or even each execution with a different test duration, but consider that from now on, when we say “test scenario” or “test execution”, we will be typically referring to one-hour tests.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Baselines and increasing scenarios
Baselines and increasing scenarios
During a performance testing project, we will spend most of the time preparing the simulation because the automation at a protocol level is very demanding and complex, but as we already said, the execution is the only moment of the project where we are adding value.
It’s very difficult to predict and plan how much time we will need to execute tests and tune the system because it depends on what we find and how hard it is to find the roots of the problems and the solutions! But something that we have to consider for sure is that we will need time to execute many tests. Here we will explain which tests and why.
Baselines
At the beginning we need to measure how much time it takes to execute a script if the whole infrastructure is dedicated for it andonly for it. That would mean the best time possible. This is called the “baseline” and it’s useful for different aspects:
- If the baseline does not reach the acceptance criteria (the expected response time) we don’t need to simulate load because we have some work to do! We need to improve the application, the test case without concurrence, because it will definitely be worse when executing many users at the same time.
- The baseline works as a metric for comparison when executing heavier loads. We will be able to say something like “with 100 concurrent users the response times increase 10% compared to the baselines”. Baselines are executed for each test case separately and many times, in order to have a statistical number. We consider that executing each test case between 10 and 30 times is good.
Incremental Scenarios
We typically name the designed load “the 100% scenario”. It’s like the whole load, the goal. If we directly execute the 100% scenario, in 99% of the cases we will get so many errors and problems that it will be very difficult to decide where we could start tuning or repairing problems. That’s why the best strategy for this is to take baby steps. We suggest starting with 20% of the load and according to the results, continue scaling the load towards the goal.
This approach is very useful because with lower loads the first problems that arise are the worst ones. Typically, these are the easiest to correct, because they are, most of the time, related to by-default configurations or environment related problems. Also, it’s a way to test our test artifacts and be sure that our test scripts behave correctly, measure correctly and do what they must do.
Even if you feel confident with the tooling, after some executions you will improve your whole tool set, from the configuration to the process of collecting and analyzing data. We are sure that you do not have to start with the whole load, you have to go step by step, resolving the indicated problem in the most adequate moment.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Want a cookie?
Want a cookie?
Let’s start with more information about how HTTP works. Remember that as HTTP is a stateless protocol, it is not possible to identify which requests come from the same user. Each request stands on its own and is unrelated to other requests. This is a problem, because typically one wants to know what a user has done before, in the previous requests.
For example, a user wants to access her or his account in a site s/he visits frequently. For that s/he has signed up and has a username and a password, so, when s/he accesses the site, the username and password are required in order to guarantee that the person accessing is the one who has signed up (there could be important information about the person, even the person’s credit card data). So, there is an HTTP request where the person sends the user and password. Let’s say for instance: after the person logged in, let’s say he or she wants to buy something on the site, so, there is another HTTP request saying that the s/he wants to buy an item using the credit card information that is associated with this account, have the item sent to a certain address also stored together with her or his user information. The application has to be completely sure that the second request corresponds to the same user. If the application cannot associate different requests from the same user, the system should ask the username and password in every request, after every user action. That would drive a person get crazy, wouldn’t it?
In order to face this problem, there is a special mechanism called cookies, small pieces of information (a short string) that are stored on the client side, and then sent to the server attached to each request. One of them is a Session identifier, useful to identify the user and its session.
When the user accesses a website, the server returns a “Set-cookie” header providing a cookie name, expiration time and the content data. When the user accesses the same website the browser returns the cookie if it hasn’t expired. This way, as the user sends her or his username and password, and in the following requests, the browser sends a cookie which lets the server realize that they correspond to the same user, all the communication between the user and the server is traceable.
Why is this important for a performance tester?
We will see that we need to simulate most tasks that the browser performs, and one of them is cookies management. Remember that we will simulate hundreds and perhaps thousands of concurrent users from the same machines, and we don’t want the server to realize that something strange is happening. Actually, we have to prepare our simulation to be as real as possible. For the server responding to the virtual users should be the same as responding to the real user, if it isn’t, our test is not as good as it should be.
To do this in the best possible way we need to understand how cookies work and how JMeter manages them, so, check the next unit!
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
How do we select good assertions?
How do we select good assertions?
JMeter, as most of the automation tools, automatically validates the response code according to what we explained before. Just to remind you, 4XX and 5XX are response codes to indicate that there was a problem on the client side or on the server side, respectively.
How do I know what to validate in the response of an HTTP request? This is an important question, because just validating a response code is not enough, we could miss other errors that we need to identify in order to act accordingly. Some websites have the bad habit of responding “There was a problem, the server is down” with a 200 response code. Or if we are automating a banking system, transferring money from one account to another, perhaps the application under test is responding “The selected account does not have enough money to process the transfer”, so the server is not processing what we planned, we would not be stressing the server as we wanted first.
Let’s see some tips in order to add great validations to our scripts.
Validation after the login
Typically, after authenticating a system, a “Welcome @username” message appears and our name appears somewhere on the screen, indicating that the login was successful. In these cases, we have an excellent element to validate.
Let’s see this with an example very common in our daily life. When we access our email account, in the homepage we are asked to provide our username and password, and once we are in, we can see our profile name at the top, our profile picture, and also some elements that help us to confirm that we have logged in successfully.
Validate window content
Many other times requests return a page in which you can choose text/titles that should come in the HTML response. For example, if we access the JMeter site, (http://jmeter.apache.org/) we can see the following:
We know that if this page loads correctly, the title “What can I do with it?” should appear in the response to the request. If it doesn’t appear, then you know you have an error, so it’s a good assertion to add as a validation that we are getting the expected page.
The challenge in this type of validation is to verify content where we know that, if there is an error, then that text does not come in the response. We can rely on developers to provide us with this kind of information.
Consider the next steps in the script
If we know that the next action to take is click a button X, or access the link Z, or select in the combo Option B, a very good strategy is to validate that the element on which we operate in the next step, appears in the response of the request. For example, if after loading the JMeter site, I want to select the option “Download Releases” I can validate this text and I assure the flow continues. For this, you can capture the response from the server verifying that it includes the option we want to check:
Take special care with special characters
Be careful when writing the assertion in JMeter because the text is not always given in the same format that is displayed. For example, to validate the title Apache JMeter ™, we cannot write it verbatim because the server response comes as Apache JMeter & trade; and not the other way as seen on the screen. If you do not use the text as it appears in the HTML, the validation will not find it.
Use a proper amount of assertions
JMeter allows us to add as many assertions as the we want, so that each of the assertions that were named before may be accompanied by other assertions such as the titles of the HTML responses (<title> … </ title> ), in order to add strength to the validation. When adding more assertions, we have to be careful that the script is not too loaded, which is why it’s not advisable to add more than two assertions per request, so we must check that the ones we choose are determining factors in defining the correctness of the response.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
Basic concepts on automation and load simulation
Basic concepts on automation and load simulation
Let’s start introducing some basic concepts that we will use along the course.
A concept we are going to use frequently is “Test Case”.
With “test case” we are referring to a sequence of user actions on the system, with the aim to follow a specific goal. Also known as test steps, test flow or test script, we prefer “a test case” as it should also include the definition of the data used, the expected results, and pre/post conditions.
When you automate a test case, you simulate being a real user. To do that, you need to capture the interaction between the application and the user. For example, when the application is a web application the interaction is made using a browser. So if we want to simulate to be a real user, we need to simulate the interaction between the browser and the system under test.
When we execute a script (automated test case) which simulates be a real user, we have a virtual user. A Virtual user is a script that acts just like a real user would when making HTTP requests to a web application.
Another important aspect is to automate the verification of the expected results. We need to asure that the automatic execution of the test steps was successfully executed, that each step was executed as expected, so the results have to be validated. In order to do that we add assertions to the scripts, which is an automatic way to validate the expected results against the actual result.
JMeter allows us to record the interaction (HTTP Requests) between the browser and the application server. Recording a test case is the first step in the process of scripting, so it is the first thing that we have to understand. In this lesson we will review the process of recording and then, in following lessons, we will learn to record a script using JMeter.
Script recording process step by step:
1. Start recording.
2. When you start recording, JMeter starts a proxy and captures the traffic between the browser and the application server.
3. Open your browser and start your test case.
4. Stop your recording.
5. Create the script based on recording.
6. Finally, web requests are generated by the script and sent to servers directly without using the browser.
Summarizing:
There is another tool that we use frequently to debug all http requests, named Fiddler. With Fiddler we can review HTTP requests but also HTTP responses, this is useful because it helps you to debug the entire traffic. While we are using JMeter, we will use Fiddler to know if the responses given by the application server are correct or not.
Next stage: Fiddler!