JMeter and Moodle logos

 

 

 

 

 

Note: this blog post is part of the Oktobertest series

This tutorial is aimed at Moodle administrators who have never used JMeter. It will help you get started but won’t teach you advanced techniques. Should you wish to dig further into this wonderful piece of open-source software, I invite you to read the following:

Before you start this tutorial, you need to download the following:

 

It is possible that you’ll get an error if you’re trying to run this in Moodle 2.2+. All you need to do is open the ‘index.php’ file (for the ‘loadtesting’ report) and search-replace all instances of the entire word ‘mod’ (no quotes) and replace it with ‘module’. This should solve your problem.

 

 

Step 1: Generate your JMeter script

  • Settings > Site Administration > Reports > JMeter loadtesting
  • Use the drop-down box to select the correct category
  • Click ‘Select category’

Note: you must be logged in as an administrator to do this

 

 

 

Most of the options above are self-explanatory, please note the following:

1. This can be a bit confusing. You should see each activity type as a different test. For example, if 10 is entered in this box, then a total of 40 virtual users would be running simultaneously (10 for chat, 10 for forum, 10 for glossary and 10 for quiz)
2. I always tick this, to put as much strain on the disk as possible, and to mimic real world usage as closely as possible
3 & 4. I like to pick and choose activities. The main reason being that students cannot post to the News forums, and also to ensure that the activities I picked are not restricted by time, or by condition (conditional activities), or even available only to particular groupings.
5. I usually tick this option, again to mimic real work usage as closely as possible
6. I like to set this to 10, for no particular good reason, I tend to like even numbers
7. Once you click the button, your Moodle server will generate a zip file, so make sure you have a zipping program installed. It will look like nothing is happening, simply go and check your download folder to see if jmeter.zip is there.

 

 

Step 2: Start JMeter

 

Step 2: Start JMeter

 
  • To open JMeter, you have to visit the JMeter folder you downloaded
  • JMeter > bin > ApacheJMeter.jar
  • Double-click on the ApacheJMeter.jar file

 

 

Step 3: Open the newly generated script

 

 
  • Select the correct file
  • Click ‘Open’

 

 

Step 4: You can now run your script

 

Step 4: You can now run your script!

 
  • Click the play button
  • Look for the numbers at the top right of the window reach the maximum set, and then go back to zero
  • Once your script has looped through the amount defined in your settings, the load test is over

Note: Check out this blog post to find out what tools you could use to monitor your server while load testing

 

 

Step 5: Check the results

 

Step 5: Check the results

 
  • There are several types of reports you can take a look at (see rectangle in picture above). 
  • Not all reports are super user-friendly and might not make a great deal of sense. Further reading on the matter:

See below for more report examples

 

 

 

 

 

 

 

 

Tip 1: Create a ‘Load testing’ category with courses dedicated to load-testing

 

  • If you are planning on enrolling lots of test users to your Moodle courses, you should know that it could have an impact on the performance of the courses involved. 
  • If you create a specific category with dedicated courses, you do not run this risk as the courses are only used once in a while. 
  • You should also reset your courses once in a while

 

 

Tip 2: Generate your script with lots of users

 

The first time you generate a script chances are you’ll have no idea how many users your installation can cope with.
Part of load testing is to find out the limits of your server. Consider this scenario:

  • Generate a script for 50 users
  • Get the script to generate users and enroll them in the course(s)
  • Run the test
  • If that test runs smoothly, you’ll have to create a new script with a new value every time you want to increase the number of users.

Consider this (better) scenario

  • Generate a script for 1,000 users (or large number of users)
  • Get the script to generate users and enroll them in the course(s)
  • Run the test with 50 users (edit the ‘Number of threads’ value)
  • If the test runs smoothly, all you need to do is edit the ‘Number of threads’ value and run the test again (the users are already enrolled)

 

 

Tip 3: Generate your script with lots of activities, and all activity types

 

  • It is possible for you to easily disable activities right in the script after it has been generated.
  • You might want to test the same course under different loads e.g. once with just a forum, once with a forum & a quiz, once with a forum, a quiz and a chat, etc.
  • By creating a test with multiple activities, you can easily re-use the same test by simply disabling/enabling the activities you want to test/not test
  • To disable an activity, you need to right-click on the activity name and then select ‘disable’.
  • All disabled activities show up in grey

 

 

Tip 4: Edit the ramp-up period

 

  • By default, the script generator sets the same value for the ‘Ramp up period’ as the number of users you selected when creating the script. 
  • The ramp-up period determines how long it takes for all virtual users (threads) to start. 
  • For example. if set to 0, all users would start at the same time. If set to 10 (as above), it takes 10 seconds to start all users (1 per second)
  • You can play with different values and monitor how your server reacts to the changes

 

 

Tip 5: Disable some of the reports

 

 

If you are running tests with a lot of virtual users and multiple activities, you might get ‘out of memory’ errors on the computer you are running JMeter. To avoid this:

  • Right-click on a specific report
  • Select ‘Disable’
  • This will not delete the report and you can decide to enable it again later
 

Average amount of RAM used for specific activities

Each new version of Moodle allows teachers to create more engaging online courses, and do so more easily as Moodle becomes more user-friendly. It also puts extra strain on the server(s) it runs on. I have been asking myself for a while: “Just how much more horsepower do each new versions of Moodle require?” October being Oktobertest for me, I thought I’d investigate further and run some tests of my own.

The results

It is important to note that while I tried my best to compare apples with apples, this test is not perfect and I am sure it could be improved. If you are interested in how I performed the tests, or to judge to what extent these findings are reliable, please scroll all the way down to ‘My Workflow’. One important point to note is that I assume that the ‘Performance information‘ provided by Moodle is accurate.

Throughout the graphs, shorter bars mean better results. You can click on the graphs to reveal full size images.

 

Each new version of Moodle uses more RAM

Average amount of RAM to generate a page

This one is hardly a surprise, but I was shocked to find out that on average Moodle 2.3 requires 3.5 times more RAM than Moodle 1.9 to generate pages – that is a lot! It is also interesting to note that Moodle 2.3 requires 13% more RAM than Moodle 2.2 and 18% more than Moodle 2.1. It will be interesting to see how Moodle 2.4 fares in this test. 

What this means to you: Make sure your Moodle server has plenty of RAM. This also means that unless you are going to run Moodle 2.x with very few concurrent users (a handful), stay away from shared hosting – it simply won’t work well enough, or you’ll end up seeing your account revoked for using too much resources.

 

It takes longer for the server to generate pages

Average time needed to generate a page

On average, it takes 3 to 4 times longer for a Moodle  2.x page to be generated by the server, compared with Moodle 1.9. While a user isn’t likely to notice the difference, it means that Moodle 2.x is more taxing on the hardware. For each Moodle page generated, the server calls files stored on the hard drive, reads/writes database –  this requires CPU and disk input/output.  

What this means to you: you will reach the limits of your hardware more quickly with Moodle 2.x. 

 

An increasing number of files are needed to generate Moodle pages

Average number of files included to generate a page

Under the hood, Moodle’s code is mainly a collection of PHP files saved in folders. With each iteration of Moodle, an increasing number of those files need to be fetched by the server for each page generated. Moodle 2.3 requires the server to fetch 3 times more files than for Moodle 1.9.

What this means to you: using a PHP accelerator for Moodle 1.9 was a recommendation, I would say that for Moodle 2.x it is a requirement. Using a PHP accelerator will save PHP files in your server’s RAM. This is important as RAM is much faster than disk access, meaning your pages will be generated more quickly.

 

Moodle 2.x is more taxing on the CPU

Average CPU load

Warning: this metric is to be taken with a pinch of salt. While I ensured as much as possible that nothing else but Moodle was running while doing the tests, it is possible that there was some background tasks happening on the server during specific tests. I decided to include this graph as I believe fluctuations would be diluted in the amount of server requests that I made.

Moodle 2.3 is almost 60% more demanding on the CPU than Moodle 1.9. Interestingly enough it seems as though Moodle 2.2 was even more demanding than Moodle 2.3, at least in this run of the test.

What this means to you: if you are running CPU demanding activities (e.g. quiz) and you are already reaching the limits of your hardware on 1.9, seriously consider upgrading your hardware before upgrading Moodle to its newest version.

 

Moodle 2.x makes more calls to the database

Average number of database reads

Average number of database writes

This was hinted in one of the points above, here is the proof. On average, Moodle 2.3 makes over twice as many calls to the database as Moodle 1.9 (reads). It is interesting to note that Moodle 2.2 made a lot more calls than Moodle 2.1 or 2.3 during my tests, seemingly due to the ‘forum’ pages. I would need to investigate further to find the cause of the problem, but it is outside the scope of this exercise.

Note that there is no ‘database writes’ data for Moodle 1.9 as it is not provided in the performance information. 

What this means to you: if you run a large Moodle site, you might want to run your database on its own server. If not, you might want to double-check that your database isn’t a bottleneck in your installation.

 

Some activities are much more demanding than others

 

Average amount of RAM used for specific activities

Moodle HQ has always made this known, some activities are much more resource hungry than others. I must admit I was expecting the quiz to be much more demanding than that. As it turns out the ‘Choice’ is a more demanding activity than the quiz.

What this means to you: if you have an ‘underpowered’ server, you might want to steer clear of some activities e.g. quiz, chat

 

Conclusions

  • Moodle 2.x is much more taxing on the hardware than Moodle 1.9 (but so much better in so many ways…)
  • If you are currently running Moodle 1.9 and are going to upgrade to Moodle 2.x, make sure you do plenty of load testing first
  • Have plenty of RAM in your server
  • Use fast disks in your Moodle server
  • Use a PHP accelerator
  • Place your database inside its own server if you run a large site
  • Follow the performance recommendations, they’re awesome
  • If you can afford it, please update to at least Moodle 2.3, your users will thank you for it

 

My Workflow

 

Server

 

Specs

  • VPS 6 cores at 2.4GHZ
  • 2GB of RAM
  • Ubuntu server 10.04LTS 
  • For those who love numbers, you can check the UnixBench results for this server here.

 

Webserver

  • Apache with the following modules loaded
    • core_module (static)
    • log_config_module (static)
    • logio_module (static)
    • mpm_prefork_module (static)
    • http_module (static)
    • so_module (static)
    • alias_module (shared)
    • auth_basic_module (shared)
    • authn_file_module (shared)
    • authz_default_module (shared)
    • authz_groupfile_module (shared)
    • authz_host_module (shared)
    • authz_user_module (shared)
    • autoindex_module (shared)
    • cgi_module (shared)
    • deflate_module (shared)
    • dir_module (shared)
    • env_module (shared)
    • expires_module (shared)
    • headers_module (shared)
    • mime_module (shared)
    • security2_module (shared)
    • negotiation_module (shared)
    • php5_module (shared)
    • reqtimeout_module (shared)
    • rewrite_module (shared)
    • setenvif_module (shared)
    • ssl_module (shared)
    • status_module (shared)
    • unique_id_module (shared)
  • Prefork
  • MySQL database
  • APC
  • Memcached

  

Moodle

 

Versions tested

I installed 4 versions of Moodle on the same server.

  • Moodle 1.9.19+
  • Moodle 2.1.8
  • Moodle 2.2.5
  • Moodle 2.3.1+

All versions of Moodle contain the same amount of courses and users. I did not install any third-party plugins and they all use the ‘Standard’ theme. I added a logo by modifying some of the core code. Authentication is set to ‘Manual accounts’ and sessions are saved on the disk.

 

Course

I created a ‘Performance Testing’ course with a selection of activities/resources. I used exactly the same course on all 4 different versions of Moodle, including blocks and filters. As mentioned above, the same users were enrolled on all 4 versions. Basically, everything was exactly the same on every version. 

Moodle performance testing course

Should you want to try and test your server as well, you can download the courses I used below:

You will also need the users for those courses, download this CSV file to batch upload 50 test users to your Moodle installation. 

 

Test strategy

 

Macros

I created 4 macros using iMacros, which mimic a single user performing the following:

  • Login > View course > View 500KB resource > Logout
  • Login > View course > View directory > Logout
  • Login > View course > Post to forum > Sends choice > Logout
  • Login > View course > Take a 7 question quiz > Logout

I ran each macro one after another 50 times for each version. Altogether, this created over 6,500 Moodle pages. Nothing else was running on the server at the time of testing.

You can download the macros I used here. You will need to change a few things for it to work for you, namely:

  • http://www.somemoodlesite.com/versionxx/ to reflect your actual Moodle installation
  • view.php?id=xxxx to be changed to your course/activities ID’s
  • responses in the macros that deal with quiz. You might want to record a macro using iMacros running the quiz and then edit the macro to check the actual values for your site

 

Recording data

It is possible for a Moodle administrator to show performance data on each page footer. I setup iMacros to scrape that data and save it in a CSV file for every page it visited. 

 

Analysing data

Rather than overcomplicating my iMacros scripts, I decided to manipulate the data post collection using TextWrangler fantastic search & replace multiple files to clean up my data so that it could be analysed in Excel. I then create pivot tables and generated graphs from those.

You can download the raw data here if you would like to play with it. I recorded how long it actually took for the pages to fully load on my computer while running the tests. I decided against mentioning this in the blog post as the metric is subject to be influenced by too many factors (server and computer were not in the same network).

 

Please use the comments sections for questions regarding this test or share similar tests you may have run.