Actions

Regression and unit tests

From LimeSurvey Manual

  Warning : NEVER run tests on a production system. The tests WILL delete and modify the limesurvey database as well as the config.php file. Always do a backup of important data before running the tests.


 Hint: Always make sure new code follows the PSR-12 standard for PHP. Use codesniffer and mess detector to catch common bugs and style fixes in your code.
 Hint: Every time you discover a new bug, you should add a regression test to make sure it does not appear again.


Since LimeSurvey 2.65.6 you have the possibility to add unit tests to LimeSurvey.

Since LImeSurvey 3.0.0, we use Selenium, headless Firefox (geckodriver) and Facebook web driver to test with the browser from PHPUnit.

Introduction

An extensive test suite will give the programmer a better sense of security, and allow him/her to make bigger re-factorizations without risking the soundness of the codebase. In an ideal world, a regression test is added every time a new bug is discovered. That way you make sure you never have to fix the same bug twice.

You can run the tests locally if you want, or you can setup a docker container that mimics the Travis CI environment. Alternatively, you can visit Travis CI after you've pushed to github to make sure your commits didn't break anything. If you think Travis is falsely reporting a bug, you should contact the person who constructed the test.

  Since LimeSurvey 5.2.15, test is only included in GIT version. You have to clone our git repo for test.


Requirements

  • A working LimeSurvey development environment
  • FireFox
  • Java Runtime Environment (JRE) or Java Development Kit (JDK) which includes JRE
  • Selenium - cross-platform Java archive .jar (tool for authoring functional tests across most modern web browsers)

https://selenium-release.storage.googleapis.com/3.7/selenium-server-standalone-3.7.1.jar

  • Gecko Driver - platform specific selenium browser driver for FireFox

https://github.com/mozilla/geckodriver/releases

Notes

PhpUnit in container

The current PhpUnit config expects Selenium to be running on the same machine as a PhpUnit process. If running the PhpUnit process from within a Docker container or a VM this might be problematic. It is possible to configure PhpUnit to connect to a selenium process on another machine. It may however be simpler to just run PhpUnit at the operating system level.

Installation

General Installation

  1. Download Selenium and the geckodriver. Extract the geckodriver.exe and place it inside the installation folder and place the selenium jar file at the same location.
  2. Create a file named "enabletests" in the installation root folder.
  3. Make sure you have Java installed, and start Selenium in a separate terminal:
    java -jar selenium-server-standalone-3.7.1.jar -enablePassThrough false
    If you want to use the headless browser make an environment variable called "MOZ_HEADLESS" and set it to 1 
    MOZ_HEADLESS=1 java -jar selenium-server-standalone-3.7.1.jar -enablePassThrough false
  4. To run tests navigate to the installation folder
  5. Now you can run tests! All tests are located inside the "tests" folder. To run a specific test:
DOMAIN=localhost PASSWORD=password ./vendor/bin/phpunit ./tests
DOMAIN=localhost PASSWORD=password ./vendor/bin/phpunit ./tests/PATH/TO/THE/FILE

Example: 

DOMAIN=localhost PASSWORD=password ./vendor/bin/phpunit ./tests/functional/backend/AdminViewsTest.php
DOMAIN=localhost PASSWORD=password ./vendor/bin/phpunit ./tests/functional/frontend

Vagrant Box Setup

  1. Install Virtualbox
  2. Install Vagrant
  3. If you have used Docker before and/or have Hyper-V activated you have to disable it, Virtualbox and Hyper-V can not run simultaneously. You can do this by executing following in powershell. 
    bcdedit /set hypervisorlaunchtype off
    If the command is denied make sure you run your powershell as system administrator: Press Windows+R to open the “Run” box. Type “powershell” into the box and then press Ctrl+Shift+Enter to run the command as an administrator.
  4. Clone the Vagrant Box into your desired folder, the Vagrant Box can be found here
    git clone https://github.com/LimeSurvey/LimeSurvey-Vagrant.git YOURFOLDERNAME
  5. Modify the file 'Vagrantfile' in the root folder and edit the IP address if it is already taken and set the PHP version to your requirements (usually 5.6 or 7.4). If you need other versions, then please look at the README file in the Repository. There are other settings like the SQL and Xdebug settings that can be customized to your needs.
    #Custom variable for installation
ip                = "10.0.0.10"
#php setup
phpversion        = "7.4" 
#mysql setup
mysqlRootPass     = "password"
mysqlDBName       = "limesurvey"
mySqlDBUser       = "limesurvey"
mySqlDBPassword   = "password"
#postgres setup
postgresDB        = "limesurvey"
postgresPassword  = "password"
#limesurvey setup
limeSurveyBranch  = "master"
#IDE KEY for Xdebug
ideKey  = "VSCODE"
  6. Remove the # before
    #enable this line on windows machine (Optional).
#vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
  7. Set the branch you want to use during Setup, by default it is set to the "master" branch 
    limeSurveyBranch  = "master"
  8. Start vagrant in the root Folder where the Vagranfile is located:
    vagrant up
    Be patient, this will take 10-15 minutes on the first start because various resources need to be downloaded and installed.
  9. You can now access your Installation by typing the hostname set in the Vagrantfile or using the ip address and appending a "/". By default this is set to "limesurvey". Follow the the normal Limesurvey Installation process. The Database username is "limesurvey" and password is "password" unless changed.
  10. Install your desired PHP version on your PC, not in the Vagrant Box
  11. Run composer installation inside your local Installation Folder in Windows "VAGRANTFOLDER/INSTALLATIONFOLDER" 
    composer install
  12. Download Selenium and the geckodriver. Extract the geckodriver.exe and place it inside the installation folder and place the selenium jar file at the same location.
  13. Create a file named "enabletests" in the installation root folder.
  14. Make sure you have Java installed, and start Selenium in a separate terminal:
    java -jar selenium-server-standalone-3.7.1.jar -enablePassThrough false
    If you want to use the headless browser make an environment variable called "MOZ_HEADLESS" and set it to 1, in the powershell:
    $Env:MOZ_HEADLESS = 1
    This will only be set in the current powershell window.

If you want to set it permanently, open the environment variables in windows and create the variable. You will have to reload the powershell window for the settings to take effect.

  1. To run tests log in to your Vagrantbox from the vagrant folder and navigate to the installation folder 
    vagrant ssh
# by default the installation folder is mapped to /var/www
cd /var/www/
  2. Now you can run tests! All tests are located inside the "tests" folder. To run a specific test:
    DOMAIN=localhost PASSWORD=password ./vendor/bin/phpunit ./tests/PATH/TO/THE/FILE
    Example: 
    DOMAIN=localhost PASSWORD=password ./vendor/bin/phpunit ./tests/functional/backend/AdminViewsTest.php
    If you should have permission issues set the synced_folder fmode option in the Vagrantfile to "fmode=777". For the new permission setting to work you must shutdown the vagrant machine and start it up again. Type 
    sudo shutdown now
    in the SSH console and after the machine has shut down, start it up again using 
    vagrant up
    as shown above.

Docker Setup

If you have a docker development environment you can use docker image provided by the Selenium project. The docker container exposes a Selenium server pre-configured with a headless Firefox browser.

https://hub.docker.com/r/selenium/standalone-firefox/tags

More information can be found on GitHub: https://github.com/SeleniumHQ/docker-selenium


Linux

(All commands below are executed in the LimeSurvey web root folder.)

The browser tests uses Firefox, so you have to install it before running.

  1. A good idea is to clear the database and make a fresh install using the command-line (backup any data you want to save first):
    DROP DATABASE limesurvey; CREATE DATABASE limesurvey

    $ DBENGINE=INNODB php application/commands/console.php install admin password TravisLS no@email.com verbose
    Make sure your config.php has the correct database credentials.
  2. To enable unit testing, you have to issue the command
    $ touch enabletests
  3. To install Facebook web driver and PHPUnit, run
    $ composer install
  4. Install node (needed for EM tests). Not used right now.
  5. Download Selenium and the geckodriver:
    $ wget "https://selenium-release.storage.googleapis.com/3.7/selenium-server-standalone-3.7.1.jar"

    $ wget "https://github.com/mozilla/geckodriver/releases/download/v0.23.0/geckodriver-v0.23.0-linux64.tar.gz"

    $ tar xvzf geckodriver-v0.23.0-linux64.tar.gz
  6. Make sure you have Java installed, and start Selenium in a separate terminal:
    $ java -jar selenium-server-standalone-3.7.1.jar -enablePassThrough false
    If you want to use the headless browser, prepend with:
    MOZ_HEADLESS=1
  7. Run tests with 
    $ DOMAIN=<local domain> ./vendor/bin/phpunit
    where "<local domain>" is replaced with your localhost domain, e.g. "localhost" or "localhost/limesurvey".

Windows

TODO

Mac

TODO

Usage

After all steps for the installation are completed, you should be able to write 

$ PASSWORD=adminpassword DOMAIN=your_limesurvey_domain phpunit

in the LimeSurvey root folder to run the tests.

If LimeSurvey is installed on your web root, use "DOMAIN=localhost".

This is an example of the output: 

 $ phpunit
 PHPUnit 5.6.2 by Sebastian Bergmann and contributors.

 ....                                                                4 / 4 (100%)

Time: 713 ms, Memory: 14.00MB

OK (4 tests, 24 assertions)

You can get some more information if you use the switch --testdox: 

$ phpunit --testdox
PHPUnit 5.6.2 by Sebastian Bergmann and contributors.

ls\tests\DateTimeForwardBack
 [x] Q 1

ls\tests\DateTime
 [x] Wrong input
 [x] Correct date format
 [x] Q 1

During testing, you might want to use --group <test group> and/or --stop-on-failure.

What can be tested?

Right now, there are tests for the expression manager, qanda and updatedb helper. The Selenium driver makes it also possible to test the full browser experience, including JavaScript.

What should be tested?

Everything that can be reported as a bug should be tested, but especially critical is of course the relation between user input and database, that is, that the correct answer is saved.

Adding a test

First of all, make sure to head over to the phpunit web site and read the introduction.

When done with that, I will assume you want to add a new test for a question type. Tests for questions are stored in folder tests/questions.

You should create a minimal survey to reproduce your bug. Export it as an lss file and put it in tests/data/surveys. Then create a class for your test. Name the file the same as the class, like "DateTimeForwardBackTest.php". The class name must end with "Test", so phpunit can find it.

Inside the class you might want to add code to import and delete the survey. I usually do this class-wide, which means that the survey gets imported when the class is loaded and then deleted when all tests in the class is done (whether failed or successfully).

The static methods setupBeforeClass and teardownAfterClass is used to setup and teardown the test fixture, respectively. You can read more about fixtures in the phpunit manual.

Let's list what we know so far: 

namespace ls\tests;

use PHPUnit\Framework\TestCase;

/**
 * @since 2017-06-16
 * @author Olle Haerstedt 
*/
class DateTimeDefaultAnswerExpressionTest extends \TestCase
{
  public static function setupBeforeClass()
  {
    // Init stuff
  }

  public static function teardownAfterClass()
  {
    // Tear down what you setup before.
  }
}

To test with Selenium and the Facebook web driver, check out the wiki page on github.

TODO

Github Actions

All tests are run automatically after each commit using Github Actions.

Cypress

New Cypress tests can be found here: https://github.com/LimeSurvey/limesurvey-core-testing

Retrieved from "http://www.limesurvey.org/manual/index.php?title=Regression_and_unit_tests&oldid=315650"