The Magento Functional Testing Framework was presented by Tom Erskine during the contribution day following MageTestFest, held in Amerfoort in November of the previous year. Currently under heavy development, it has already seen 2 releases and has reached version 2.0.3.
Why do we need to perform functional testing?
Considering the complexity of the Magento 2 architecture, manually testing everything during the acceptance testing phase becomes very resource-consuming. You can have multiple different flows for each feature (and these flows could be quite complex). Commonly, as new features are added over time, the amount of time your QA team spends on tests will increase.
Also humans make mistakes all the time, especially when they have to follow these complex flows it’s easy to get distracted and do something wrong. Even though you can cover all the code with unit and integration tests – they don’t guarantee that everything will work correctly in different browsers and on different devices. That is why a QA team should extensively use automatic tests, as it is the only viable solution to fully test the product. It’s much faster, less error-prone and along with test results you will also get extra data for free: like how long each test was running, on which device, etc. The developer will know exactly what went wrong, he won’t need to ask QA for additional details, which happens sometimes when the developer can’t reproduce the issue. In the end, automatic testing should save a lot of time and money for everyone.
Tests in XML
The fundamental difference in the MFTF approach is its use of XML to define tests. There was a lot of buzz on the internet regarding this topic because it is very controversial. Usually when someone says that you need to define tests in XML – one imagines a gigantic unmaintainable XML file. But as you’ll see later with MFTF it’s not the case and resulting tests are quite small and properly structured. Another possible benefit is that one can write some XML processing tools to automatically modify/transform or generate tests. With XML this is much easier than with PHP since the XML structure is much easier to process. Also visually these tests look quite easy to read comparing to PHP code. Another quite important benefit is that a QA engineer, who will write tests should mostly know XML and HTML.
Main Reasons to Use MFTF
So, what are the main reasons to choose MFTF over other functional testing frameworks?
For one it’s quite important that it is based on a well-established Codeception testing framework. It gives its users some guarantees of stability and security for their commitment to MFTF.
Second reason is that it is built BY Magento and FOR Magento. That means that we should get the tool optimized for our specific tasks. Also Magento actively listens to community so feel free to reach them out if you miss something in the MFTF.
Third reason is scalability. When writing tests for MFTF you can reuse parts of other tests, section and page definitions, which allows you to hide the overall complexity of the project and this also gives you the ability to split the work load easily among several QA engineers. That is the scalability of development process. MFTF also gives you infrastructure scalability as you can easily execute different groups of tests in different environments and run them in parallel.
Fourth reason is support for XML automatic processing which I’ve already mentioned. It for example can be used to automatically generate the test project from some template.
The last reason is that you can use services such as BrowserStack to test your project on thousands of browsers running on real devices.
Environment setup
Before writing any tests, we first need to prepare our environment. This process is well-described in the official documentation for the MFTF. So please follow the instructions from there. Basically you just need to install Selenium, Chrome driver and Allure CLI which is easy to do on a Mac with the help of the Brew package manager. After that you need to disable secret keys in admin URLs and disable WYSIWYG editor. Download the Chrome driver to the same directory as selenium-server and start the selenium-server. Now we can move to writing the test.
A simple MFTF test for the Magento Search module
First in this test we need to check that the search is working and magento 2 search page shows correct result. To check this we need to perform the search for a specific product and check that it is present in the search results. So we need to create this product somehow, then perform the search and after the test we need to delete the product from the catalog. Let’s review it using a simple Magento 2 MFTF test example
To create a product we need to use the createData action which essentially sends a POST request to the Magento REST API. You need to pass entity parameter to it so it will know what to create. You can see that this action has stepKey set to searchableProduct and this key will receive the result of this action which we will use in our test later.
<before>
<createData entity="SearchableProduct" stepKey="searchableProduct"/>
<amOnPage url="{{HomePage.url}}" stepKey="amOnPage"/>
</before>
The result of the createData action will be a data object, which will contain all the field values of the created product (at least those which are returned by Magento REST API) and you can use any of these fields to perform assertions or to do whatever you need with them.
After we have created our test product let’s make sure that we are on a home page. We can do this using amOnPage action. We just need to pass url parameter to it (along with stepKey, of course).
As you’ve noticed, I’ve put these two actions in the before block which, hence the name, will be executed before the test.
In the beginning of the test you need to put some annotations which will be used for reporting and for running the test. Please, at least define title, description and a test group. Make sure to give meaningful titles and descriptions so the person who will read the report will know what this test does exactly. You can define many tests in one group and run them all at once. And any test can belong to multiple groups – just define multiple group elements in the annotations section.
<annotations>
<title value="Simple Search Test"/>
<description value="Creates a simple product, checks that it
displays in the search results"/>
<group value="search"/>
</annotations>
Now that we have our product and annotations we can actually start writing the test. As you remember we already pointed the browser to the home page so now we just need to fill in the search query and submit the form. Let’s use fillField and click actions for that. For both of them we need to supply element selectors, which we should define in SearchFormSection (we will get back later to it).
Once the form is submitted we need to wait for the page to load and there is a waitForPageLoad action for that. Also, because I’ve heard a few people asking, I would like to point out that you can use IDE autocompletion feature to find actions and action parameters – this works very well with MFTF.
So now that we are on a search results page we need to verify that there is our product and the search page title is correct. For this lets use see action. It takes selector and userInput as parameters. In userInput you specify the value we are looking for. If this action won’t see userInput in the specified element the test will fail.
With the last action we check that search results page header is correct and contains a text “Search results for: ‘product name’”. That’s all for the actual test.
<fillField stepKey="enterSearchQuery"
selector="{{SearchFormSection.query}}" userInput="$
$searchableProduct.name$"/>
<click stepKey="clickSearchButton"
selector="{{SearchFormSection.searchButton}}"/>
<waitForPageLoad stepKey="waitPageLoad"/>
<see stepKey="checkFirstProductTitle"
selector="{{SearchResultsSection.firstProductTitle}}" userInput="$
$searchableProduct.name$"/>
<see stepKey="checkPageTitle"
selector="{{SearchResultsSection.pageTitle}}" userInput=“Search results
for: '$searchableProduct.name$'"/>
Now that we are done with testing, lets remove our fixture using deleteData action. All you need to do is pass createDataKey to it. Then it will send DELETE request to the Magento REST API and the product will be deleted.
<after>
<deleteData createDataKey="searchableProduct"stepKey="deleteProduct"/>
</after>
Sections and data definitions
Here you see that I’ve defined two sections SearchFormSection and SearchResultsSection. These definitions are pretty much self-explanatory. You just put all the elements you need in each section, specify name of the element, type and selector for this element. That’s all.
<section name="SearchFormSection">
<element name="query" type="input" selector="#search"/>
<element name="searchButton" type="button"selector="#search_mini_formbutton[type=submit]"/>
</section>
<section name="SearchResultsSection">
<element name="pageTitle" type="text" selector="h1.page-title"/>
<element name="firstProductTitle" type="text"selector=".product-items>.product-item:first-child a.product-item-link"/>
</section>
And the last thing is our SearchableProduct data object. You see that it has its type attribute set to product and this type is actually defined in the Catalog module. It maps this type to a specific REST API URL for each type of actions: create, update and delete. So if you need to create your own fixtures – you can – just take a look at definitions inside Catalog/Metadata/. This will give you an idea of how to do that.
Other than that – this data object definition is easy to understand. You just define all the fields you need to set for this product (keep in mind, that the values you put in there should be acceptable by Magento REST API).
<entity name="SearchableProduct" type="product">
<data key="sku" unique="suffix">SearchableProduct</data>
<data key="type_id">simple</data>
<data key="attribute_set_id">4</data>
<data key="name" unique="suffix">SearchableProduct</data>
<data key="price">123.00</data>
<data key="visibility">4</data>
<data key="status">1</data>
<data key="qty">123</data>
<required-entity type="product_extension_attribute">EavStockItem</required-entity>
</entity>
Running the tests
There are two ways to run the tests: using robo or using codeception. It’s just a matter of preference. With robo you need to type less, with codeception on the other hand you have more flexibility. The robo itself invokes codeception and prints the exact command it executes. So you can just copy-paste it and modify it to your needs if you are debugging something.
The Robo way:
robo group searchThe Codeception way:
./vendor/bin/codecept run functional --verbose --steps --skip-group skip --group search
Once you’ve run the tests, you can generate the test report. MFTF generates reports using Allure which is flexible and lightweight multi-language test reporting tool. You’ll just need to run these two commands to open the report in your browser:
vendor/bin/robo allure2:generate
vendor/bin/robo allure2:open
The resulting report has a nice UI and cool chart animations. So digging through the test reports should be quite an enjoyable experience.
All these things can easily be integrated into continuous integration pipeline and tests could be performed each time you commit your changes to repository or on some schedule. This will allow your team to be notified about all the issues just right when they appear.
Registering custom modules
If you want to develop your modules in a custom namespace, then you need to include full paths to your modules in the .env file:
CUSTOM_MODULE_PATHS=/Users/username/some/path/Company/SomeModule/
You can put here multiple modules, just separate them using commas.
Note: hopefully MFTF developers will come up with a better approach of adding your custom modules to the project.
Creating your own type of fixtures
There are two things you need to do to define your own fixture type:
1. Create REST API endpoint which will handle create/update/delete operations for your type.
2. Define your type in the Metadata directory of your test module.
We’ll skip the first and I will show a sample definition of the type:
<operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/etc/dataOperation.xsd">
<operation name="CreateSomething" dataType="something" type="create" url="/V1/yourmodule/something/{id}" method="POST" auth="anonymous">
<contentType>application/json</contentType>
<object dataType="something" key="item">
<field key="id">integer</field>
</object>
</operation>
<operation name="DeleteSomething" dataType="something" type=“delete" url="/V1/yourmodule/something/{id}" method="DELETE" auth="anonymous">
<contentType>application/json</contentType>
</operation>
<operation name="UpdateSomething" dataType="something" type="update" url="/V1/yourmodule/something/{id}" method=“PUT" auth="anonymous">
<contentType>application/json</contentType>
<object dataType="something" key="item">
<field key="id">integer</field>
</object>
</operation>
</operations>
There are some limitations though:
Only one parameter is replaced in the REST API url (see {id} above) operation name should be a camel case of operation type + your type name. As a reference you can use Catalog module – there are a lot of type definitions.
Conclusion
MFTF is already a ready-to-use tool, although it lacks some core tests for Magento. But you can build tests for your projects based on it and there are some companies who already do that. If you have some feedback or need help with it – feel free to share it/ask for help on Magento Community Engineering Slack channel (magentocommeng.slack.com) with MFTF developers.
This post was written for Atwix blog by Stanislav Smovdorenko, Full-Stack Developer & Founder of Mobelop. Stanislav likes solving hard problems and making things run fast.