Stress/Load-Testing of Asynchronous HTTP/REST Services with JMeter

Home  >>  Common  >>  Stress/Load-Testing of Asynchronous HTTP/REST Services with JMeter

Stress/Load-Testing of Asynchronous HTTP/REST Services with JMeter

On November 28, 2011, Posted by , In Common, By ,,,,, , With 31 Comments

Although I have been using JMeter for stress- and load-testing of web applications a good few times it took us a while to figure out how to test asynchronous HTTP/REST based services with the tool. With us I mean a fellow programmer – Holger Staudacher, I have the honor to work with currently on a project – and my humble self.

While Holger developed restfuse based on the experience of doing functional and integration tests for the project mentioned above, we decided to use JMeter for stress- and load-testing. The main service of the software under test processes a data structure that is uploaded to a certain URL. If the upload process was successful an URL pointing to a resource containing the processing result is returned. The resulting resource is not available immediately – processing takes a while. So polling is used to retrieve the resource once it is available1.

Our goal was to measure the time it takes to upload the data structure, process it and download the result resource in one test run. Running such a test with multiple users concurrently should give us a fair impression of the throughput capabilities of the system. Does not sound too complicated, but…

…our first approach writing a test plan for the scenario described in the previous paragraph using the on-board capabilities of JMeter did not work out very well. Neither was the plan comprehendible nor – and that was worse – made the measurement results any sense. In particular clamping the upload request and the polling loops together with a transaction controller seemed to have some unexpected side effects with timers. So after a while of additional Google research I stumbled across the JavaSamplerClient API, which I did not know before.

See also  Performance Analysis of REST/HTTP Services with JMeter and Yourkit

There is an entry at stackoverflow.com that describes how to extend the AbstractJavaSamplerClient – an implementation of JavaSamplerClient – and use it within JMeter. So this was the way to solve our problem. We created an extension of AbstractJavaSamplerClient overriding runTest(JavaSamplerContext). Within that method we use HttpClient to perform the upload and poll requests. Once the processing result is successfully retrieved by a poll request all the header and content information is stored in an instance of SampleResult. The latter is returned by the overridden test sampler method for further processing by JMeter – quite straight forward2.

Once you have created a jar that contains a custom JavaSampleClient and put this into the lib/ext/ folder below your JMeter installation directory you can add a Sampler type Java Request to your Thread Group. This allows you to select and configure a custom sampler as shown in the picture below:

Using the the JavaSamplerClient made our test plan very simple and allowed us to use the common JMeter result measurement functionality as shown below examplary with the Graph Results view:

And of course the measurement results are now reasonable…

Since we had to fumble quite a while to get this done I thought our solution might be of interest for other people, too – which was the reason to write this post. But it would be also interesting to hear from you if there are easier solutions we did not notice. So feel welcome to provide feedback :-)


Frank Appel
Follow me
Latest posts by Frank Appel (see all)

31 Comments so far:

  1. Jason Weden says:

    Check out this analysis I just found today related to this: https://github.com/excilys/gatling/wiki/Benchmarks

  2. Gayatri says:

    What would you suggest for unit testing rest API – restfuse, or Jmeter?

    • Frank Appel says:

      Your question already contains the answer. For REST API testing I would use restfuse. JMeter comes into play if you want to stress-/load-test your application.

      • Anil says:

        Hello Frank,

        I too have similar requirement. can you please share the code yo have written.

        • Frank Appel says:

          I am sorry, but the post is based on work for a customer which is not open source. However given the description above it should not be that difficult to implement it by yourself.

  3. Benamar says:

    Hello Frank,

    You said = ” The latter is returned by the overridden test sampler method for further processing”

    How return you the results of multiple requests in your runTest?

    thanks.

    • Frank Appel says:

      Benamar,

      I don’t know if I understand your question correctly. But if I remember it right the whole point of a sampler is to work on single request basis. Multiple requests will be executed by JMeter subsequently in a loop or with multiple threads concurrently. But for each request a single delegation call to the sampler is issued by JMeter. Therefore you actually don’t have to deal with multiple results in the sampler implementation.

      Regards
      Frank

    • Bappa says:

      Can a sample be shared please. @Anil

    • Deepak says:

      Can you share the code ?

  4. Benamar says:

    Thank you for your quick response.
    I used a thread pool to execute the simultaneous requests in ‘runTest’ method , and I used result.addSubresult (result [i]) to aggregate the results.

    I can see the requests results in the results tree (as subresults) but the are absents in the reports!

    Do you have any solution to get the results of these simultaneous tests in reports ?

    Thank you.

    • Frank Appel says:

      I’m sorry but I don’t have a solution for this right at hand as I do not remember working with subresults.

  5. DK says:

    I have a similar requirement but we are using WebHooks.
    So, once the request is processed our REST Service will POST a response back to the callback URL sent in the request.

    Can this be stress tested with JMeter using a similar approach?

    • Frank Appel says:

      Hm – I never did it, but I could imagine that it is possible to implement an AbstractJavaSamplerClient that serves as “server” for the callback URL. Such an implementation would block (something like ServerSocket#accept() instead of poll like described above) until the callback URL is triggered. But that’s just a spontaneous idea which sounds a bit low level at first glance – maybe there is a better solution around somewhere…

  6. Shashank Pandey says:

    Can you please share the code for accepting the requests and sending a reponse. I just need a generic code for accepting a Student object. In my scenario the callbacks are giving me a jaxb object via a PUT body. I want to write that object in a file or on console. Please help (specially the connecting and returning the object part).

    • Frank Appel says:

      I am sorry, but as I already mentioned to anil the post is based on work for a customer that is not open source. So I am not able to post any part of it here.

  7. Shashank Pandey says:

    And even the callbacks are being accepted via a rest interface

  8. lizzyenpeter.be says:

    Hi, just wanted to mention, I loved this blog post. It was helpful. Keep on posting!

  9. rinzenzandro says:

    I created a new project, exported it to a jar, put the jar in lib/ext of jmeter but I don’t see my class in “Classname” inside “Java Request” in jmeter.
    what am I doing wrong?
    does my jar need to have a special name? special package structure?
    do I need to rename my jar to ApacheJMeter_java.jar?
    do I need to add my class to that jar and recompile?

    • Frank Appel says:

      There is no need a for special name or package structure, as far as I know. The jar must contain your compiled custom JavaSampleClient of course. But otherwise I don’t know about any particularities.

  10. Mandar says:

    I want to load test a Restful service which accepts XML file as an input parameter. Do you have any idea how I can use JMeter to load test it? Or how we can pass XML to Restful Webservice?
    Regards
    Mandar

    • Rüdiger Herrmann says:

      Did you have a look at the JMeter documentation (search for ‘xml’, the fourth or so hit)? It says that there is a ‘Post Body’ tab on the HTTP Request configuration panel that you can use to post raw content.

      HTH
      Rüdiger

  11. Niharika says:

    Hey Frank
    I have a situation in my project and we are using JMeter for performance testing.
    We have RESTFul API and I need to load test these services.
    The intent of this test is to reach the threshold. This will require me to increase the user load at runtime gradually until the application breaks down.
    Could you please suggest how to go about this?
    So far I have been doing all the configuration in the script and then execute it. I have no clue how can I make JMeter to do this at runtime.
    Any help would be appreciated.
    Thanks
    Niharika

    • Frank Appel says:

      Hi Niharika,

      did you try to configure your thread group with an appropriate amount of threads and ramp up time for continously increasing the load? For more information on this you might have a look at Test Plan manual description, section 4.1 ThreadGroup.

      Hope that helps
      Frank

  12. Niharika says:

    Yes Frank
    I generally do it for 5-10 threads for a duration of 1-2 hrs. Also used a constant throughput timer to stick to the throughput per SLA.
    So this sums up to 100k transactions, which my application is capable of handling.
    I want to test beyond these numbers and execute a test, where I can increase user load at runtime if I find the application does good.
    Was I able to explain my concern any better?
    Let me know if you know of any other tool which might help. I will do the research.
    Thanks

    • Frank Appel says:

      Now I think I understand. You want to change your test-plan configuration while it is executed. Well, I am not aware of a possibility to do that with JMeter (but that must not mean much…). However, why not running the test-plan multiply times with different load? Doing this in a way like a binary search algorithm, you should find the capacity threshold of you application reasonably fast.

      I mean run you app with a large load that crashes it pretty soon. For the next run decrease the load by half. If the app crashes, decrease load again by half else increase it to the middle of the current and the previous load. After a few iterations you should get a good impression what load your application can bare. This might work out even faster than looking for a tool. Good luck!

  13. Rhubarb says:

    you answered a comment about restfuse vs JMeter saying that it depends on what you want to test. But wouldn’t you be using restfuse test cases in your JMeter plans to set up the REST calls?

    • Frank Appel says:

      Probably I do not understand your question correctly, but restfuse does not integrate with JMeter out of the box. And I am not sure if this would be reasonable anyway, as both tools handle the http client requests in their own way. However there might be a way of combining them and maybe this could be interesting in a way that one can actually perform stress/load tests on JUnit tests defined with restfuse.

  14. Dmitri T says:

    Perhaps using JSR223 sampler with the same code could be more easy as it wouldn’t be necessary to develop custom extension and it’ll allow to keep the code either on file system or in JMeter test plan itself making it easier to change on-the-fly. See How to Load Test AJAX/XHR Enabled Sites With JMeter for an example of using JSR223 sampler for sending asynchronous requests overriding parent thread group limitations.

  15. majid says:

    Hi,

    I need a tool for stressing a Rest Service with almost 2000 requests per second. I tried Jmeter and set number of threads to 2000 but it seems Jmeter can not handle this many connections simultaneously. Do you know if it can be ever done with Jmeter? If not do you know an alternative tool?

    • Rüdiger Herrmann says:

      Maybe the limitating factor here is the VM and/or (network) hardware and not JMeter. Did you do some research or ask the JMeter mailing list for advice? I am sure others solved similar problems before…