Manual tests

Yes, we have "manual tests" in something called "integration tests". These actually are integration tests (we're testing a bundled system, not independent components), they're simply not automated for various reasons: complexity of implementation of the automation vs resources availability vs maintainability. Hopefully some of these tests will be more easily automatable once we revamp the GUI components.

These tests should be executed at least once per release. The procedure to execute them is described at
Release QA Plan#Manualtestsusingtheintegrationtestswebapp.

To add such manual tests:

• checkout http://svn.magnolia-cms.com/svn/community/bundle/trunk/
• depending on what needs to be tested, you can do a combination of:
  • add templates in magnolia-integration-tests-module/src/main/resources/mgnl-files/templates/test/manual
  • declare those templates in magnolia-integration-tests-module/src/main/resources/info/magnolia/test/config.properties or via info.magnolia.test.SetupStuffForTests.
  • in the template itself, describe what actions need to be performed for the test to be considered successful
  • add a page with this template to the website, under /manual-tests, via magnolia-integration-tests-module/src/main/resources/info/magnolia/test/website.properties.

please take this into consideration if you find yourself creating many templates that have no purpose other than holding the test description: in the future, the current website.properties file might be specialized (i.e use a custom, simpler, format, instead of the properties import format; additionally, the test description might figure in there instead of the template, especially for tests where templating isn't relevant); if you're in such case, feel free to do so yourself or shout at Greg !

Automated tests

We also have a suite of automated tests. They rely on the same process, but are executed differently:

• since magnolia-integration-tests lies inside the CE bundle project, they are executed every time to bundle is built. This means they're executed on a regular basis by Hudson (http://hudson.magnolia-cms.com/job/magnolia-bundle_trunk/) and every time we do a release.

To add automated tests, the procedure is similar to the above, but:

• if you add website pages, you'll want to add them under /testpages
• you need to write code to execute the test; the current tests reside in magnolia-integration-tests/tests/src/test. info.magnolia.integrationtests.AbstractMagnoliaIntegrationTest is meant to be a base class such tests could extend.

There are also groovy scripts in magnolia-integration-tests/tests/src/main/:

• Testing of activation that activates content and checks for results
• Crawling test that requests all the pages and checks for rendering exceptions
• Log crawling test that reads logs and checks for rendering exceptions.

Concurrent builds for bundles on Jenkins

Integration tests run concurrently on Jenkins for CE and EE bundles, and for our various branches (5.0.x up to master). Therefore each branch/bundle must use its own ports for starting up Jetty and e.g. sending shutdown signals through the Cargo wrapper.
Here are the ports in use for our current builds:    

Configuration for these ports is located in:

• magnolia-integration-tests/tests/pom.xml , properties <containerHttpPort>  and <cargoRmiPort>
  • automated tests, manual tests for IntelliJ IDEA
• info.magnolia.testframework.AbstractMagnoliaIntegrationTest => DEFAULT_DOMAIN
  
  • manual tests for eclipse

 When branching a bundle's master to the next major version, one should update these ports according to the pattern in the table above. 

Debugging

To debug these tests, one way to go is to start the container the same way as for the manual tests, with mvn clean verify -Pmanual-tests, and run the tests from the IDE in debug mode.
Somehow, memory issues seem to arise, at least with EE, in the manual-tests mode. This seemed to have helped, not sure which statement exactly:

export JAVA_OPTS="-Xms64M -Xmx1024M -Djava.awt.headless=true"
export JAVA_OPTIONS="-Xms64M -Xmx1024M -Djava.awt.headless=true"
export JETTY_OPTIONS="-Xms64M -Xmx1024M -Djava.awt.headless=true"
export JETTY_OPTS="-Xms64M -Xmx1024M -Djava.awt.headless=true"

Generalities

All these tests are executed within the context of a single Jetty instance, which is started by the Cargo plugin of Maven. The current setup deploys two Magnolia instances (magnolia-test-webapp and magnolia-test-webapp-public, under the /magnoliaTest and /magnoliaTestPublic contexts, respectively) which currently both are an extension of magnolia-empty-webapp magnolia-module-samples magnolia-integration-tests-module, which is containing the pages, templates etc created above, which are used while executing the tests.

If more instances or different setups would be needed, we would probably have to extend this rather simple architecture quite a bit, at which point it will probably become unpractical to run these within the bundle; we'll have to figure out a procedure to ensure those tests are still systematically executed WITH the release being made.