How to configure and start working with Magento Test Automation Framework

Magento TAF is a pretty good thing to automize your Magento store testing. When you start working with Magento TAF, you have many tests, which are ready for use, so this framework can help to develop all kinds of tests without building an automatic infrastructure. Since a TAF users guide is a bit outdated, and some things in the framework have been already changed we’ve decided to write an article on this topic.

At the base level, Magento TAF will require the following software:
PHP: 5.2.0 or later
PHPUnit: 3.5 or later
Java SE Development Kit
Apache
MySQL Server
Selenium RC 2.0.3

Furthermore, Magento TAF contains the following two configuration files:

1. config.yml – contains the settings of the Selenium Server.
It is located in:
../magento-afw-x.x.x/config/config.yml

Here is a default content:

## Set up browsers for running tests  ##                         

// List of browsers which can be used
browsers:
    googlechrome: &googlechrome
         name: 'Google Chrome'

// Browser name
         browser: '*googlechrome'

// Host where it is installed and will be run
         host: 'localhost'

// Port on the Host where this browser will be available
         port: 4444

         timeout: 60
    firefox: &firefox
         name: 'Firefox'
         browser: '*chrome'
         host: 'localhost'
         port: 4444
         timeout: 60
    iexplorer: &iexplorer
         name: 'Internet Explorer'
         browser: '*iexplore'
         host: 'localhost'
         port: 4444
         timeout: 60

// Default browser for run tests
    default: *firefox

## Framework configuration  ##      

framework:

// Base pathes to the fixture and testsuite
    fixture_base_path: 'fixture'
    testsuite_base_path: 'testsuite'

    // work only if tests are run for single browser
    shareSession: true
    coverageScriptUrl: ''

  // Capture screenshot on failure works only for browsers that support it: Firefox/Chrome, IE with snapsIE tool
    captureScreenshotOnFailure: true

// Parameter sets up the saves HTML pages on failure
    saveHtmlPageOnFailure: false

    load_all_uimaps: true
    load_all_data: true
    cache:
        frontend:
            name: 'core'
            options:
                caching: false
                cache_id_prefix: 'selenium_'
                automatic_serialization: true
        backend:
            name: 'file'
            options:
                cache_dir: 'var/cache'

##  Applications for running tests   ##

//List of Applications which can be tested

applications:

//Settings of Magento  application

// Name of  Magento application
    mage: &mage

        fallbackOrderFixture: 'default'
        fallbackOrderHelper: 'Core'
        areas:

// General link, base UImap, login and password to the backend
            admin:
                url: 'http://www.localhost.com/magento/admin/'
                base_page_uimap: 'log_in_to_admin'
                uimap_path: 'admin'
                login: 'admin_login'
                password: 'admin_password'

// General link, base UImap, login and password to the frontend
            frontend:
                url: 'http://www.localhost.com/magento/'
                base_page_uimap: 'home_page'
                uimap_path: 'frontend'
                login: ''
                password: ''

// General link, base UImap, login and password to the paypal sandbox           
 paypal_sandbox: &paypalSandbox
                url: 'https://www.sandbox.paypal.com/'
                base_page_uimap: 'paypal_sandbox'
                uimap_path: 'paypal_sandbox'
                login: 'paypal_sandbox_login'
                password: 'paypal_sandbox_password'

// General link, base UImap, login and password to the paypal developer
            paypal_developer: &paypalDeveloper
                url: 'https://developer.paypal.com/'
                base_page_uimap: 'paypal_developer_home'
                uimap_path: 'paypal_developer'
                login: 'paypal_developer_login'
                password: 'paypal_developer_password'

// Default Magento application which can be used
    default: *mage

2. phpunit.xml – contains the list of the test-scripts (test cases) to be run, and it is located in:
../magento-afw-x.x.x/phpunit.xml.

Default content:

<phpunit backupGlobals="true"
         backupStaticAttributes="false"
         bootstrap="bootstrap.php"
         cacheTokens="true"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         forceCoversAnnotation="false"
         mapTestClassNameToCoveredClassName="false"
         printerClass="PHPUnit_TextUI_ResultPrinter"
         processIsolation="false"
         stopOnError="false"
         stopOnFailure="false"
         stopOnIncomplete="false"
         stopOnSkipped="false"
         testSuiteLoaderClass="PHPUnit_Runner_StandardTestSuiteLoader"
         strict="false"
         verbose="true">
    <testsuites>
        <testsuite name="Test Suite">
            <directory suffix=".php">tests/Customer/</directory>
        </testsuite>
    </testsuites>
    <logging>
        <log type="json" target="./var/logs/logfile.json"/>
        <log type="tap" target="./var/logs/logfile.tap"/>
        <log type="junit" target="./var/logs/logfile.xml" logIncompleteSkipped="false"/>
        <log type="testdox-html" target="./var/logs/testdox.html"/>
        <log type="testdox-text" target="./var/logs/testdox.txt"/>
    </logging>
</phpunit>

Then, for executing all tests, you should change the test suite name to All Tests, choose directory with all tests, and leave empty test name in the phpunit.xml file, like on the following example:

   <testsuites>
        <testsuite name="All Tests">
            <directory suffix=".php">testsuite</directory>
        </testsuite>
    </testsuites>

Moreover, for executing one or few test suites, you should change a test suite name to Test Suite, choose the directories with test suites and leave empty test name. If you want to execute all tests from the suite, it is necessary to enter the name of a single test from the chosen directory:

<testsuites>
        <testsuite name="Test Suite">
            <directory suffix="RegisterTest.php">testsuite/core/Mage/Customer/</directory>
 <directory suffix=".php">testsuite/core/mage/Newsletter/</directory>
        </testsuite>
    </testsuites>

For executing a single test, you should change a test suite name to Single Test, enter full name of the test file and full path to this file:

  <testsuites>
        <testsuite name="Single Test">
            <directory suffix="SimpleTest.php">testsuite/core/Mage/Product/Create/</directory>
        </testsuite>
    </testsuites>

In case of executing Magento TAF – follow these steps:

1. Run Selenium Server.

$ java -jar BaseDir/selenium-server-standalone-x.xx.x.jar

2. Execute runtest.bat (in Windows) through the command prompt for test running.

3. Execute runtest.sh for Linux/*NIX OS through the terminal for the running.

$ sh BaseDir/runtests.sh

After executing runtests.sh in the terminal, you will be able to see the current progress.
When all the tests have been executed, you’ll see a description of all failures, errors, incompletes, skips.

Legend:

“.” – test PASSed
“S” – test SKIPped
“I” – test INCOMPLETE
“F” – test FAILed
“E” – error appears during test running

Configuration read from /Users/admin/taf/magento-afw/phpunit.xml

..FF.........................................F.................  63 / 241 ( 26%)
....................F.............F............................ 126 / 241 ( 52%)
..........F...............F.................................F.F 189 / 241 ( 78%)
..............................................FFFFFE

Time: 20.13 hours, Memory: 36.00Mb

There was 1 error:

1) Core_Mage_Product_DuplicateTest::duplicateConfigurable
OutOfRangeException: Cannot find page with mca "catalog_product/edit/attributes/NjQ1/set/4/type/configurable" in "admin" area

/Users/admin/taf/magento-afw/framework/Mage/Selenium/Helper/Uimap.php:301
/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:1697
/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:1733
/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:2876
/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:2857
/Users/admin/taf/magento-afw/tests/Product/Helper.php:429
/Users/admin/taf/magento-afw/tests/Product/DuplicateTest.php:306

--

There were 14 failures:

1) Core_Mage_Product_Create_BundleTest::allFieldsForDynamic
general_news_from: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
general_news_to: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
prices_special_price_from: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
prices_special_price_to: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
design_active_from: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')
design_active_to: The stored value is not equal to specified: ('1/8/14' != '1/8/2014')

/Users/admin/taf/magento-afw/framework/Mage/Selenium/TestCase.php:1384
/Users/admin/taf/magento-afw/tests/Product/Helper.php:618
/Users/admin/taf/magento-afw/tests/Product/Create/BundleTest.php:123

That’s all for now. We hope this article will help you to start working with the test automation framework and you are welcome to share your experience and thoughts about it in the comments.

Cookie	Duration	Description
__jid	1 day	Used to add comments to the website and remember the user's Disqus login credentials across websites that use said service.
cf_ob_info	past	The cf_ob_info cookie is set by Cloudflare to provide information on HTTP Status Code returned by the origin web server, the Ray ID of the original failed request and the data center serving the traffic.
cf_use_ob	past	Cloudflare sets this cookie to improve page load times and to disallow any security restrictions based on the visitor's IP address.
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
li_gc	2 years	Linkeding cookie that stores the user's cookie consent state for the current domain
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Cookie	Duration	Description
_ga	2 years	Google Analytics cookie. Used to identify unique users and expires after two years
_gat	1 day	Used by Google Analytics to throttle request rate
_gid	24 hours	Google Analytics cookie. Used to identify user and expires in 24 hours.
_hjAbsoluteSessionInProgress	30 minutes	Hotjar cookie. Used to detect the first pageview session of a user. 30 minutes duration. Boolean true/false data type.
_hjFirstSeen	User session	Hotjar cookie. Identifies a new user’s first session. Used by Recording filters to identify new user sessions. Session duration. Boolean true/false data type.
_hjIncludedInPageviewSample	30 minutes	Hotjar cookie. Set to determine if a user is included in the data sampling defined by your site's pageview limit. 30 minutes duration. Boolean true/false data type.
_hjIncludedInSessionSample	30 minutes	Hotjar cookie. Set to determine if a user is included in the data sampling defined by your site's daily session limit. 30 minutes duration. Boolean true/false data type.
_hjRecordingLastActivity	User session	Hotjar cookie. Sets a unique ID for the session. This allows the website to obtain data on visitor behaviour for statistical purposes
_hjSession_2881021	30 minutes	Hotjar cookie. Holds current session data. Ensures subsequent requests in the session window are attributed to the same session. 30 minutes duration. JSON data type.
_hjSessionRejected	User session	Hotjar cookie. If present, set to '1' for the duration of a user's session, when Hotjar has rejected the session from connecting to our WebSocket due to server overload. Applied in extremely rare situations to prevent severe performance issues. Session duration. Boolean true/false data type.
_hjSessionUser_2881021	1 year	Hotjar cookie. Set when a user first lands on a page. Persists the Hotjar User ID which is unique to that site. Ensures data from subsequent visits to the same site are attributed to the same user ID. 365 days duration. JSON data type.
_hjTLDTest	User session	Hotjar cookie. Registers statistical data on users' behaviour on the website. Used for internal analytics by the website operator.
AnalyticsSyncHistory	29 days	Linkedin cookie. Used in connection with data-synchronization with third-party analysis service.
disqus_unique	1 year	Disqus cookie. Collects statistics related to the user's visits to the website, such as number of visits, average time spent on the website and loaded pages.
https://#.#/	User session	leadfeeder.com cookie. Registers statistical data on users' behaviour on the website. Used for internal analytics by the website operator.
juggler/event.gif	User session	Disqus cookie. Identifies the visitor across devices and visits, in order to optimize the chat-box function on the website.

Cookie	Duration	Description
aet-dismiss	User session	Disqus cookie. Necessary for the functionality of the website's comments system
drafts.queue		Disqus cookie. Necessary for the functionality of the website's comments system
lang		Linkedin cookie. Remembers the user's selected language version of a website
submitted_posts_cache	User session	Disqus cookie. Necessary for the functionality of the website's comments system

Cookie	Duration	Description
_hjRecordingEnabled	User session	Hotjar cookie. This cookie is used to identify the visitor and optimize ad-relevance by collecting visitor data from multiple websites – this exchange of visitor data is normally provided by a third-party data-center or ad-exchange.
_lfa	2 years	Leadfeeder cookie. Used to store and track audience reach and expires in 2 years.
_lfa_expiry	persistent	Leadfeeder cookie. : Contains the expiry-date for the cookie with corresponding name
_lfa_test_cookie_stored	1 day	Leadfeeder cookie. Used in context with Account-Based-Marketing (ABM). The cookie registers data such as IP-addresses, time spent on the website and page requests for the visit. This is used for retargeting of multiple users rooting from the same IP-addresses. ABM usually facilitates B2B marketing purposes
ads/ga-audiences	User session	Used by Google AdWords to re-engage visitors that are likely to convert to customers based on the visitor's online behaviour across websites.
badges-message	Persistent	Disqus cookie. Used for comments functionality
bcookie	2 years	Used by the social networking service, LinkedIn, for tracking the use of embedded services.
bscookie	2 years	Used by the social networking service, LinkedIn, for tracking the use of embedded services.
lidc	1 day	Used by the social networking service, LinkedIn, for tracking the use of embedded services.
UserMatchHistory	29 days	Linkedin cookie. Ensures visitor browsing-security by preventing cross-site request forgery. This cookie is essential for the security of the website and visitor.

How to configure and start working with Magento Test Automation Framework

Subscribe to our newsletter.