How to configure and start working with Magento Test Automation Framework

Magento TAF is a pretty good thing to automize your Magento store testing. When you start working with Magento TAF, you have many tests, which are ready for use, so this framework can help to develop all kinds of tests without building an automatic infrastructure. Since a TAF users guide is a bit outdated, and some things in the framework have been already changed we’ve decided to write an article on this topic.

  • At the base level, Magento TAF will require the following software:
  • PHP: 5.2.0 or later
  • PHPUnit: 3.5 or later
  • Java SE Development Kit
  • Apache
  • MySQL Server
  • Selenium RC 2.0.3

Furthermore, Magento TAF contains the following two configuration files:

1. config.yml  – contains the settings of the Selenium Server.
It is located in:
../magento-afw-x.x.x/config/config.yml

Here is a default content:

## Set up browsers for running tests  ##                         

// List of browsers which can be used
browsers:
    googlechrome: &googlechrome
         name: 'Google Chrome'

// Browser name
         browser: '*googlechrome'

// Host where it is installed and will be run
         host: 'localhost'

// Port on the Host where this browser will be available
         port: 4444

         timeout: 60
    firefox: &firefox
         name: 'Firefox'
         browser: '*chrome'
         host: 'localhost'
         port: 4444
         timeout: 60
    iexplorer: &iexplorer
         name: 'Internet Explorer'
         browser: '*iexplore'
         host: 'localhost'
         port: 4444
         timeout: 60

// Default browser for run tests
    default: *firefox

## Framework configuration  ##      

framework:

// Base pathes to the fixture and testsuite
    fixture_base_path: 'fixture'
    testsuite_base_path: 'testsuite'

    // work only if tests are run for single browser
    shareSession: true
    coverageScriptUrl: ''

  // Capture screenshot on failure works only for browsers that support it: Firefox/Chrome, IE with snapsIE tool
    captureScreenshotOnFailure: true

// Parameter sets up the saves HTML pages on failure
    saveHtmlPageOnFailure: false

    load_all_uimaps: true
    load_all_data: true
    cache:
        frontend:
            name: 'core'
            options:
                caching: false
                cache_id_prefix: 'selenium_'
                automatic_serialization: true
        backend:
            name: 'file'
            options:
                cache_dir: 'var/cache'

##  Applications for running tests   ##

//List of Applications which can be tested

applications:

//Settings of Magento  application

// Name of  Magento application
    mage: &mage

        fallbackOrderFixture: 'default'
        fallbackOrderHelper: 'Core'
        areas:

// General link, base UImap, login and password to the backend
            admin:
                url: 'http://www.localhost.com/magento/admin/'
                base_page_uimap: 'log_in_to_admin'
                uimap_path: 'admin'
                login: 'admin_login'
                password: 'admin_password'

// General link, base UImap, login and password to the frontend
            frontend:
                url: 'http://www.localhost.com/magento/'
                base_page_uimap: 'home_page'
                uimap_path: 'frontend'
                login: ''
                password: ''

// General link, base UImap, login and password to the paypal sandbox           
 paypal_sandbox: &paypalSandbox
                url: 'https://www.sandbox.paypal.com/'
                base_page_uimap: 'paypal_sandbox'
                uimap_path: 'paypal_sandbox'
                login: 'paypal_sandbox_login'
                password: 'paypal_sandbox_password'

// General link, base UImap, login and password to the paypal developer
            paypal_developer: &paypalDeveloper
                url: 'https://developer.paypal.com/'
                base_page_uimap: 'paypal_developer_home'
                uimap_path: 'paypal_developer'
                login: 'paypal_developer_login'
                password: 'paypal_developer_password'

// Default Magento application which can be used
    default: *mage

2. phpunit.xml – contains the list of the test-scripts (test cases) to be run, and it is located in:
../magento-afw-x.x.x/phpunit.xml.

Default content:

<phpunit backupGlobals="true"
         backupStaticAttributes="false"
         bootstrap="bootstrap.php"
         cacheTokens="true"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         forceCoversAnnotation="false"
         mapTestClassNameToCoveredClassName="false"
         printerClass="PHPUnit_TextUI_ResultPrinter"
         processIsolation="false"
         stopOnError="false"
         stopOnFailure="false"
         stopOnIncomplete="false"
         stopOnSkipped="false"
         testSuiteLoaderClass="PHPUnit_Runner_StandardTestSuiteLoader"
         strict="false"
         verbose="true">
    <testsuites>
        <testsuite name="Test Suite">
            <directory suffix=".php">tests/Customer/</directory>
        </testsuite>
    </testsuites>
    <logging>
        <log type="json" target="./var/logs/logfile.json"/>
        <log type="tap" target="./var/logs/logfile.tap"/>
        <log type="junit" target="./var/logs/logfile.xml" logIncompleteSkipped="false"/>
        <log type="testdox-html" target="./var/logs/testdox.html"/>
        <log type="testdox-text" target="./var/logs/testdox.txt"/>
    </logging>
</phpunit>

Then, for executing all tests, you should change the test suite name to All Tests, choose directory with all tests, and leave empty test name in the phpunit.xml file, like on the following example:

   <testsuites>
        <testsuite name="All Tests">
            <directory suffix=".php">testsuite</directory>
        </testsuite>
    </testsuites>

Moreover, for executing one or few test suites, you should change a test suite name to Test Suite, choose the directories with test suites and leave empty test name. If you want to execute all tests from the suite, it is necessary to enter the name of a single test from the chosen directory:

<testsuites>
        <testsuite name="Test Suite">
            <directory suffix="RegisterTest.php">testsuite/core/Mage/Customer/</directory>
 <directory suffix=".php">testsuite/core/mage/Newsletter/</directory>
        </testsuite>
    </testsuites>

For executing a single test, you should change a test suite name to Single Test, enter full name of the test file and full path to this file:

  <testsuites>
        <testsuite name="Single Test">
            <directory suffix="SimpleTest.php">testsuite/core/Mage/Product/Create/</directory>
        </testsuite>
    </testsuites>

 
In case of executing Magento TAF – follow these steps:

1. Run Selenium Server.

$ java -jar BaseDir/selenium-server-standalone-x.xx.x.jar

2. Execute runtest.bat (in Windows) through the command prompt for test running.

3. Execute runtest.sh for Linux/*NIX OS through the terminal for the running.

$ sh BaseDir/runtests.sh

 
After executing runtests.sh in the terminal, you will be able to see the current progress.
When all the tests have been executed, you’ll see a description of all failures, errors, incompletes, skips.

Legend:

  • “.” – test PASSed
  • “S” – test SKIPped
  • “I” – test INCOMPLETE
  • “F” – test FAILed
  • “E” – error appears during test running
Configuration read from /Users/admin/taf/magento-afw/phpunit.xml

..FF.........................................F.................  63 / 241 ( 26%)
....................F.............F............................ 126 / 241 ( 52%)
..........F...............F.................................F.F 189 / 241 ( 78%)
..............................................FFFFFE

Time: 20.13 hours, Memory: 36.00Mb

There was 1 error:

1) Core_Mage_Product_DuplicateTest::duplicateConfigurable
OutOfRangeException: Cannot find page with mca "catalog_product/edit/attributes/NjQ1/set/4/type/configurable" in "admin" area

/Users/admin/taf/magento-afw/framework/Mage/Selenium/Helper/Uimap.php:301
/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:1697
/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:1733
/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:2876
/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:2857
/Users/admin/taf/magento-afw/tests/Product/Helper.php:429
/Users/admin/taf/magento-afw/tests/Product/DuplicateTest.php:306

--

There were 14 failures:

1) Core_Mage_Product_Create_BundleTest::allFieldsForDynamic
general_news_from: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
general_news_to: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
prices_special_price_from: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
prices_special_price_to: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
design_active_from: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
design_active_to: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')

/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:1384
/Users/admin/taf/magento-afw/tests/Product/Helper.php:618
/Users/admin/taf/magento-afw/tests/Product/Create/BundleTest.php:123

That’s all for now. We hope this article will help you to start working with the test automation framework and you are welcome to share your experience and thoughts about it in the comments.

Clients

Smart Brands Choose Us.

From Mark Cuban and Sir Richard Branson backed startups, to Inc 500 US Fastest Growing Companies and Global Brands choose us to deliver and support unparalleled eСommerce experience for their customers.