Average amount of RAM used for specific activities

Each new version of Moodle allows teachers to create more engaging online courses, and do so more easily as Moodle becomes more user-friendly. It also puts extra strain on the server(s) it runs on. I have been asking myself for a while: “Just how much more horsepower do each new versions of Moodle require?” October being Oktobertest for me, I thought I’d investigate further and run some tests of my own.

The results

It is important to note that while I tried my best to compare apples with apples, this test is not perfect and I am sure it could be improved. If you are interested in how I performed the tests, or to judge to what extent these findings are reliable, please scroll all the way down to ‘My Workflow’. One important point to note is that I assume that the ‘Performance information‘ provided by Moodle is accurate.

Throughout the graphs, shorter bars mean better results. You can click on the graphs to reveal full size images.

 

Each new version of Moodle uses more RAM

Average amount of RAM to generate a page

This one is hardly a surprise, but I was shocked to find out that on average Moodle 2.3 requires 3.5 times more RAM than Moodle 1.9 to generate pages – that is a lot! It is also interesting to note that Moodle 2.3 requires 13% more RAM than Moodle 2.2 and 18% more than Moodle 2.1. It will be interesting to see how Moodle 2.4 fares in this test. 

What this means to you: Make sure your Moodle server has plenty of RAM. This also means that unless you are going to run Moodle 2.x with very few concurrent users (a handful), stay away from shared hosting – it simply won’t work well enough, or you’ll end up seeing your account revoked for using too much resources.

 

It takes longer for the server to generate pages

Average time needed to generate a page

On average, it takes 3 to 4 times longer for a Moodle  2.x page to be generated by the server, compared with Moodle 1.9. While a user isn’t likely to notice the difference, it means that Moodle 2.x is more taxing on the hardware. For each Moodle page generated, the server calls files stored on the hard drive, reads/writes database –  this requires CPU and disk input/output.  

What this means to you: you will reach the limits of your hardware more quickly with Moodle 2.x. 

 

An increasing number of files are needed to generate Moodle pages

Average number of files included to generate a page

Under the hood, Moodle’s code is mainly a collection of PHP files saved in folders. With each iteration of Moodle, an increasing number of those files need to be fetched by the server for each page generated. Moodle 2.3 requires the server to fetch 3 times more files than for Moodle 1.9.

What this means to you: using a PHP accelerator for Moodle 1.9 was a recommendation, I would say that for Moodle 2.x it is a requirement. Using a PHP accelerator will save PHP files in your server’s RAM. This is important as RAM is much faster than disk access, meaning your pages will be generated more quickly.

 

Moodle 2.x is more taxing on the CPU

Average CPU load

Warning: this metric is to be taken with a pinch of salt. While I ensured as much as possible that nothing else but Moodle was running while doing the tests, it is possible that there was some background tasks happening on the server during specific tests. I decided to include this graph as I believe fluctuations would be diluted in the amount of server requests that I made.

Moodle 2.3 is almost 60% more demanding on the CPU than Moodle 1.9. Interestingly enough it seems as though Moodle 2.2 was even more demanding than Moodle 2.3, at least in this run of the test.

What this means to you: if you are running CPU demanding activities (e.g. quiz) and you are already reaching the limits of your hardware on 1.9, seriously consider upgrading your hardware before upgrading Moodle to its newest version.

 

Moodle 2.x makes more calls to the database

Average number of database reads

Average number of database writes

This was hinted in one of the points above, here is the proof. On average, Moodle 2.3 makes over twice as many calls to the database as Moodle 1.9 (reads). It is interesting to note that Moodle 2.2 made a lot more calls than Moodle 2.1 or 2.3 during my tests, seemingly due to the ‘forum’ pages. I would need to investigate further to find the cause of the problem, but it is outside the scope of this exercise.

Note that there is no ‘database writes’ data for Moodle 1.9 as it is not provided in the performance information. 

What this means to you: if you run a large Moodle site, you might want to run your database on its own server. If not, you might want to double-check that your database isn’t a bottleneck in your installation.

 

Some activities are much more demanding than others

 

Average amount of RAM used for specific activities

Moodle HQ has always made this known, some activities are much more resource hungry than others. I must admit I was expecting the quiz to be much more demanding than that. As it turns out the ‘Choice’ is a more demanding activity than the quiz.

What this means to you: if you have an ‘underpowered’ server, you might want to steer clear of some activities e.g. quiz, chat

 

Conclusions

  • Moodle 2.x is much more taxing on the hardware than Moodle 1.9 (but so much better in so many ways…)
  • If you are currently running Moodle 1.9 and are going to upgrade to Moodle 2.x, make sure you do plenty of load testing first
  • Have plenty of RAM in your server
  • Use fast disks in your Moodle server
  • Use a PHP accelerator
  • Place your database inside its own server if you run a large site
  • Follow the performance recommendations, they’re awesome
  • If you can afford it, please update to at least Moodle 2.3, your users will thank you for it

 

My Workflow

 

Server

 

Specs

  • VPS 6 cores at 2.4GHZ
  • 2GB of RAM
  • Ubuntu server 10.04LTS 
  • For those who love numbers, you can check the UnixBench results for this server here.

 

Webserver

  • Apache with the following modules loaded
    • core_module (static)
    • log_config_module (static)
    • logio_module (static)
    • mpm_prefork_module (static)
    • http_module (static)
    • so_module (static)
    • alias_module (shared)
    • auth_basic_module (shared)
    • authn_file_module (shared)
    • authz_default_module (shared)
    • authz_groupfile_module (shared)
    • authz_host_module (shared)
    • authz_user_module (shared)
    • autoindex_module (shared)
    • cgi_module (shared)
    • deflate_module (shared)
    • dir_module (shared)
    • env_module (shared)
    • expires_module (shared)
    • headers_module (shared)
    • mime_module (shared)
    • security2_module (shared)
    • negotiation_module (shared)
    • php5_module (shared)
    • reqtimeout_module (shared)
    • rewrite_module (shared)
    • setenvif_module (shared)
    • ssl_module (shared)
    • status_module (shared)
    • unique_id_module (shared)
  • Prefork
  • MySQL database
  • APC
  • Memcached

  

Moodle

 

Versions tested

I installed 4 versions of Moodle on the same server.

  • Moodle 1.9.19+
  • Moodle 2.1.8
  • Moodle 2.2.5
  • Moodle 2.3.1+

All versions of Moodle contain the same amount of courses and users. I did not install any third-party plugins and they all use the ‘Standard’ theme. I added a logo by modifying some of the core code. Authentication is set to ‘Manual accounts’ and sessions are saved on the disk.

 

Course

I created a ‘Performance Testing’ course with a selection of activities/resources. I used exactly the same course on all 4 different versions of Moodle, including blocks and filters. As mentioned above, the same users were enrolled on all 4 versions. Basically, everything was exactly the same on every version. 

Moodle performance testing course

Should you want to try and test your server as well, you can download the courses I used below:

You will also need the users for those courses, download this CSV file to batch upload 50 test users to your Moodle installation. 

 

Test strategy

 

Macros

I created 4 macros using iMacros, which mimic a single user performing the following:

  • Login > View course > View 500KB resource > Logout
  • Login > View course > View directory > Logout
  • Login > View course > Post to forum > Sends choice > Logout
  • Login > View course > Take a 7 question quiz > Logout

I ran each macro one after another 50 times for each version. Altogether, this created over 6,500 Moodle pages. Nothing else was running on the server at the time of testing.

You can download the macros I used here. You will need to change a few things for it to work for you, namely:

  • http://www.somemoodlesite.com/versionxx/ to reflect your actual Moodle installation
  • view.php?id=xxxx to be changed to your course/activities ID’s
  • responses in the macros that deal with quiz. You might want to record a macro using iMacros running the quiz and then edit the macro to check the actual values for your site

 

Recording data

It is possible for a Moodle administrator to show performance data on each page footer. I setup iMacros to scrape that data and save it in a CSV file for every page it visited. 

 

Analysing data

Rather than overcomplicating my iMacros scripts, I decided to manipulate the data post collection using TextWrangler fantastic search & replace multiple files to clean up my data so that it could be analysed in Excel. I then create pivot tables and generated graphs from those.

You can download the raw data here if you would like to play with it. I recorded how long it actually took for the pages to fully load on my computer while running the tests. I decided against mentioning this in the blog post as the metric is subject to be influenced by too many factors (server and computer were not in the same network).

 

Please use the comments sections for questions regarding this test or share similar tests you may have run.

Free load testing tools for Moodle

[pulledquote]Moodle is becoming increasingly popular at my current school. While this is great, the server is starting to feel sluggish. I wanted to find out how much further I can take my current hardware and how much strain Moodle actually place on my server. This is why I have decided that October would be… Oktobertest for Moodle! Here are 3 ways that will help you find out if your server can cope with the demands you place/want to place on it.[/pulledquote]

Note: This is by no means supposed to be a definitive guide to load testing. It is an introduction to what can be done to ensure a Moodle installation can cope with a specific number of simultaneous users. It is aimed at Moodle admins who haven’t really done load testing before. Be careful, load testing can crash your server and you could get in trouble with your host company – always check with them first.

 

Why is Moodle so demanding? A look under the hood

Warning – what follows is a very simplified version of the truth, but close enough for jazz.

Moodle is a very powerful, but demanding web application. It is basically a collection of files saved on a hard drive and data from a database that get pieced together into a webpage for every click a user performs. For example, if a student visits a Moodle homepage, then a course page, and then posts a forum discussion, a total of 7 webpages will have been created (homepage > login page > course page > forum page > new post > post discussion > return to course page). For every page that is created, it is necessary for the server to go and collect the correct files, call the database multiple times, and also ‘write’ data to the database. The more users visit the site, the more pages need to be generated, the more strain on the server. If you follow the simple example above for a class of 30 students,  210 pages need to be generated in a very short period of time. It is also important to know that not all parts of Moodle were born equal and some of Moodle’s popular modules, such as ‘Quiz’, put more of a strain on the server than other modules such as ‘page’ or ‘URL’; This can be a real problem if your server cannot cope with a class full of students taking a test at the same time.

 

Why you should test

In short, to prevent alienating teachers and students. A lot of people use Moodle for work/school related activities, at my school alone over 800 people. I am prepared to take a risk here and say that most of those people would rather do something else than be on Moodle. As a Moodle administrator, I thrive to ensure the server is as fast and reliable as possible so that people can go about their business swiftly, and this is made possible by testing different scenarios, with specific numbers of users. I test to find out:

  • how long it takes to perform a specific task, for one or multiple concurrent users 
  • how many people can use Moodle at the same time to perform a series of tasks, and still have a usable (read fast) system
  • how much RAM, CPU, bandwidth, etc. is used during a test, and see if I am close to the limits
  • if I need to upgrade my hardware/hosting package and when I might need to upgrade

 

What to test

As you are testing your Moodle installation, you will be recording the vitals of your server, such as CPU usage, RAM, disk input/output, bandwidth, SQL usage, etc. to zero in on possible bottlenecks (check this useful blog post on server monitoring). To get meaningful results from your testing, you’ll want to have a testing strategy that mimics real world usage as much as possible. In order to achieve this, you might want to try and have a look at your Moodle logs. Here are a few rules that I try and stick to when testing:

  • test a mixture of activities
    • some that are database intensive e.g. chat
    • some that are disk intensive e.g. download large files
    • some that are CPU intensive e.g. quiz
  • add randomness to the test, if the tool allows to do so. For example apply some random ‘waiting’ time between clicks
  • test a mixture of accounts (teacher, admin, student) – not always able to do this
  • if on a ‘live’ server, perform the tests when there is little to no use e.g. 3AM for me
  • repeat the same test several times, at different times of the day. This is especially important if you are testing on a ‘live’ server. 

 

Tools to test Moodle

This post offers only a short introduction to the load testing tools and I have listed them in order of difficulty to use.

 


 

Tool 1: iMacros extension

iMacros for Firefox 

iMacros is an automation tool provided free of charge by iOpus. It is not meant to be a load-testing tool but works fine as one. There are free add-ons for Internet Explorer, Firefox and Google Chrome. I personally use the Firefox version, as it’s available both for Windows, Mac and Linux, and is more powerful than the Chrome version. 

iMacros records your web browsing activity so that you can later simulate the actions of a real Moodle user, all automatically. For example, you start up Firefox and set iMacros to record. You go about your daily business on Moodle as normal (e.g. login, view your course, add an assignment, answer some forum posts and logout), iMacros keeps a record of everything you click on during your session, including all the forms that you fill in until you press ‘Stop recording’. Those steps are saved in a macro, which you can then play back later by opening that macro. This means that you can press the ‘play’ button and Firefox will repeat all of the steps that you did during your session, without any more interactions from your part, thus automatically simulating the web browsing activities of a Moodle user. You can ‘loop’ the action to repeat it automatically as many times as you like.

I use iMacros to record simple tasks, for example:

  • Login > View course > View resource > Logout
  • Login > View course > View directory > Logout
  • Login > View course > Post to forum > Choice > Logout
  • Login > View course > Take quiz > Logout
iMacros is meant to only simulate one user at a time, which is hardly load-testing. There is a simple workaround, which is to start multiple instances of Firefox (Ctrl+N for Windows, Cmd+N for Macs) on your PC, you are then only limited by your computer’s ability to run multiple instances of Firefox (RAM, CPU and bandwitdh being the three biggest issues). Each instance of Firefox then simulates one user. On my computer, I can safely run approximately 25 instances of Firefox concurrently (please note that using different tabs will not work). If you have access to a large network of computers, you can save the macros on multiple computers and schedule them to run at set times, automatically using a task scheduler or cron.
 
Difficulty: ★
 
Pros:
  • Very easy to use for simple workflows (e.g. login, view course)
  • Gentle learning curve
  • Exact replica of what a real user’s browsing experience is like
  • Excellent support in the form of a vibrant community and wiki
  • Macros are portable and can be saved/run on multiple computers, from within or outside your network
Cons:
  • More demanding workflows require to dive into the code
  • Limited number of simultaneous users, unless macros are run from multiple computers
 
Tutorial:
 

 

Tool 2: Loadstorm

Loadstorm.com 

Loadstorm is a web based load testing service. Their business is to simulate multiple virtual users simultaneously navigating a website, following a pre-recorded scenario. It is similar to iMacros in that you can record your web activity to play later (scenario), but you have to use Loadstorm’s interface to do it. Although it is not as intuitive as iMacros, it is still simple enough to use to record simple workflows. Loadstorm is free to use for up to 25 simultaneous virtual users (forever, not just once), which is quite handy considering it is quite close to a ‘regular’ classroom size. The 25 user-limit is usually enough to test sites on shared hosting, and bring the server down. You can purchase extra virtual users to test larger sites. To date, I have not yet needed to purchase additional users as Loadstorm often runs promotions whereby you get free users if you follow them on Twitter, or like them on Facebook. I must say that their service is top-notch and will be purchasing when I run out of users.

 

Difficulty: ★★

Pros:

  • Can be used on any device with a web browser and access to the Internet
  • Easy to use for simple workflows
  • Company behind the product provides excellent support
  • Servers based around the World, mimics real-world usage for students accessing Moodle from home
  • Relatively gentle learning curve
Cons:
  • Pay-for service if you want to test more than 25 concurrent users
  • Some more complex workflows are difficult to set up (e.g. multiple quizzes)
  • Doesn’t work behind some firewalls
  • Web based service so there is some extra latency involved
 
Tutorial: 

  

Tool 3: JMeter

Apache JMeter 

JMeter is a fully-fledged load and performance open-source software by the Apache foundation. It is much more powerful than the other tools mentioned (although I suspect Loadstorm is actually based on JMeter) but also has a much steeper learning curve. Luckily, an excellent JMeter script generator was created and shared by the good people of the Open University UK (download it here). Without this tool, it would be rather difficult to create scripts (scenarios) to test your Moodle installation. This tool will help you generate users, and get those users to post to forums, participate in chat sessions and take quizzes. Nothing stops you from adding to the generated script, should you want to add more steps to it. Should you wish to dig further into JMeter, I would strongly recommend you to read the excellent ‘Apache JMeter‘ book by Emily H. Halili.

 

Difficulty: ★★★

Pros:

  • Java based, so can be used on all platforms
  • Scripts are portable – test from within or outside your network
  • Extremely powerful
Cons:

What tools do you use to load-test Moodle? Please feel free to share in the comments section.