Location
Our ui tests are placed in the ce-bundle/magnolia-integration-tests/tests because all the required setup (install and start an author and public instance) is already there.
Running the tests
Caution
Some of those tests are very sensitive - when running them on your local machine, make sure to not do anything else while they're running. e.g. don't switch to any application or type anything. This can be enough to make some of the UI tests fail.
You need docker installed as well as both the tomcat container as well as the browser for the ui-tests are started as docker containers. (For macOS have a look here)
Automated run
We use Selenium for testing our ui. The UITest will be part of magnolia-integration-test/tests and use its test-webapp & test-public-webapp. UITests are only triggered if you specify the corresponding profile (uitest).
In short, use the following command to automatically run the ui tests
launch uitests
..../magnolia-integration-tests/tests mvn clean install -P ui-tests -DseleniumBrowser=chrome
Run only the tests in one class
launch uitests for one class
..../magnolia-integration-tests/tests mvn clean install -P ui-tests -DseleniumBrowser=chrome -DfailIfNoTests=false -Dit.test=MyTestClass
(Note that this will also trigger the crawling after the test, it would be nice to figure out a command that runs one class, but does not trigger the crawling.)
Run only one test
launch uitests for one test
..../magnolia-integration-tests/tests mvn clean install -P ui-tests -DseleniumBrowser=chrome -DfailIfNoTests=false -Dit.test=MyTestClass#myTest
Manual run
If you want to run the ui tests manually from within your IDE you can start the author and public tests instance with
launch uitests
..../magnolia-integration-tests/tests mvn clean install -P manual-tests
Once this is running, then you can simply run or debug with JUnit in your IDE as you would a normal unit test.
Outlook
- Execute tests with different browsers (Firefox, Chrome, Safari, Ie, ...) an different OS's (OSX, Unix, Windows, iOs, ...) simulating different devices (e.g. ipad as well).
Implementation hints
Our UI tests are implemented with selenium. Despite the fact that this tool is really mature, those tests aren't as reproducible as ordinary unit-tests. Here's a list of hints that should help to write stable magnolia ui tests:
| Issue | Potential fix | Remark |
|---|---|---|
| Element cannot be found although it's there | waitUntil methods. | querying for an element (AbstractMagnoliaUITest#getElementByPath(By)) will explicitly try for 5 seconds - if the test triggers a long running action (e.g. activation) this can take even longer so we might have to add an additional, explicit delay |
| Element is found although it should be gone | waitUntil methods. | unlike in the case where we check for existence of an element we don't have any implicit or explicit delay here - if the element needs some time to go away (e.g. Overlay fadeout) we have to add an explicit delay. use waitUntil(elementIsGone() |
| Input field value cannot be queried with xpath | dont use xpath | input[@class = 'classname' and @value = 'form input...'] could be changed to input[@class = 'classname] and use a condition like waitUntil(attributeToBe(locator, "value", "form input...")) to query the input value. |
| Form validation fails, even if fields are properly entered | ensure blur / change | after filling an input with sendKeys, one should explicitly blur the field - i.e. click anywhere else - and allow some time for the change event to occur. Only then, pressing 'save changes' will be properly aware of the modifications. |
| Getting another element instead of the expected one | scope XPath queries | making dead simple queries like //input[@class='v-textfield'] should be carefully considered, there may be more elements of same kind (inputs, buttons) currently loaded in the UI. Try to scope selectors at least to subApp level. |
| Querying for descendent elements | use .// XPath prefix | invoking parent.findElement(By.xpath("//something")) doesn't query only for sub-elements of parent. To evaluate an XPath expression relative to parent WebElement, one needs to use the .// prefix instead. |
If you find it hard to create XPath queries, you might find the following helpful:
Firefox console can evaluate XPath expressions, through the$x()functionFirebug + FirePath (depending on Firebug), which makes it easy to test xpath statementsFirefox XPath Checker extension
If writing an XPath for the element you're looking is hard, XPath should probably not be used for this case.
Screenshots of each stage of the test
Our framework is rigged to generate a screenshot and save it during the test - this can help debug what went wrong with the test. These images are stored in tests/target/surefire-reports/
Even on hudson you can access these screenshots:
Overview
Content Tools