In this post I want to show you how to use Jenkins to automatically execute Selenium tests written in JUnit format, and how the results from these tests can be directly reported back to Jenkins. To achieve this, we need to complete the following steps:
- Write some Selenium tests in JUnit format that we want to execute
- Create a build file that runs these tests and writes the reports to disk
- Set up a Jenkins job that runs these tests and interprets the results
Note: First of all a point of attention: I couldn’t get this to work while Jenkins was installed as a Windows service. This has something to do with Jenkins opening browser windows and subsequently not having suitable permissions to access sites and handle Selenium calls. I solved this by starting Jenkins ‘by hand’ by downloading the .war file from the Jenkins site and running it using java -jar jenkins.war
Creating Selenium tests to run
First, we need to have some tests that we would like to run. I’ve created three short tests in JUnit-format, where one has an intentional error for demonstration purposes – it’s good practice to see if any test defects actually show up in Jenkins! Using the JUnit-format implies that tests can be run independently, so there can’t be any dependencies between tests. My test class looks like this (I’ve removed two tests and all import statements for brevity):
package com.ontestautomation.selenium.ci;
public class SeleniumCITest {
static WebDriver driver;
@Before
public void setup() {
driver = new FirefoxDriver();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
}
@Test
public void successfulLoginLogout() {
driver.get("http://parabank.parasoft.com");
Assert.assertEquals(driver.getTitle(), "ParaBank | Welcome | Online Banking");
driver.findElement(By.name("username")).sendKeys("john");
driver.findElement(By.name("password")).sendKeys("demo");
driver.findElement(By.cssSelector("input[value='Log In']")).click();
Assert.assertEquals(driver.getTitle(), "ParaBank | Accounts Overview");
driver.findElement(By.linkText("Log Out")).click();
Assert.assertEquals(driver.getTitle(), "ParaBank | Welcome | Online Banking");
}
@After
public void teardown() {
driver.quit();
}
}
Pretty straightforward, but good enough.
Creating a build file to run tests automatically
Now to create a build file to run our tests automatically. I used Ant for this, but Maven should work as well. I had Eclipse generate an Ant build-file for me, then changed it to allow Jenkins to run the tests as well. In my case, I only needed to change the location of the imports (the Selenium and the JUnit .jar files) to a location where Jenkins could find them:
<path id="seleniumCI.classpath">
<pathelement location="bin"/>
<pathelement location="C:/libs/selenium-server-standalone-2.44.0.jar"/>
<pathelement location="C:/libs/junit-4.11.jar"/>
</path>
Note that I ran my tests on my own system, so in this case it’s OK to use absolute paths to the .jar files, but it’s by no means good practice to do so! It’s better to use paths relative to your Jenkins workspace, so tests and projects are transferable and can be run on any system without having to change the build.xml.
Actual test execution is a matter of using the junit and junitreport tasks:
<target name="SeleniumCITest">
<mkdir dir="${junit.output.dir}"/>
<junit fork="yes" printsummary="withOutAndErr">
<formatter type="xml"/>
<test name="com.ontestautomation.selenium.ci.SeleniumCITest" todir="${junit.output.dir}"/>
<classpath refid="seleniumCI.classpath"/>
<bootclasspath>
<path refid="run.SeleniumCITest (1).bootclasspath"/>
</bootclasspath>
</junit>
</target>
<target name="junitreport">
<junitreport todir="${junit.output.dir}">
<fileset dir="${junit.output.dir}">
<include name="TEST-*.xml"/>
</fileset>
<report format="frames" todir="${junit.output.dir}"/>
</junitreport>
</target>
This is generated automatically when you create your build.xml using Eclipse, by the way.
Running your tests through Jenkins
The final step is setting up a Jenkins job that simply calls the correct Ant target in a build step:
After tests have been run, Jenkins should pick up the JUnit test results from the folder specified in the junitreport task in the Ant build.xml:
If everything is set up correctly, you should now be able to run your tests through Jenkins and have the results displayed:
You can also view details on the individual test results by clicking on the error message:
The Eclipse project I have used for this example can be downloaded here.
Scripts are running on jenkins but driver is not opened
could you tell me the how to open the driver in jenkins
What does the console output for your job look like?
Can you see any events in the Event Viewer (in case you’re running this under Windows)?
I installed Jenkins as a Windows service. Managed to get the JUnit test running, but fail to load the report:
Started by user anonymous
Building on master in workspace C:\Program Files (x86)\Jenkins\jobs\SeleniumTest\workspace
[SeleniumTest] $ cmd.exe /C ‘”C:\eclipse\plugins\org.apache.ant_1.9.2.v201404171502\bin\ant.bat -file build.xml FFAboutUsFooter && exit %%ERRORLEVEL%%”‘
Buildfile: C:\Users\shahir\workspace\SeleniumTest\build.xml
FFAboutUsFooter:
[junit] Running FFAboutUsFooter
[junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.635 sec
[junit] Output:
[junit] —-Starting the test for About Us (footer) page—-
[junit] Page title is: | Homeware | Furniture | Fashion Accessories
[junit] Clicked on Log in button
[junit] Page title is: Login | Homeware | Furniture | Fashion Accessories
[junit] Entered email: shtest1@mailinator.com
[junit] Entered password: XXXXX
[junit] Unchecked the Remember Me checkbox: false
[junit] Clicked on: Log In
[junit] Clicked on: About Us
[junit] Breadcrumb: Company Profile About Us
[junit] Hovered on: My
[junit] Clicked on: Log Out
[junit] —-The test for About Us (footer) page ended—-
[junit]
BUILD SUCCESSFUL
Total time: 52 seconds
Recording test results
ERROR: Publisher hudson.tasks.junit.JUnitResultArchiver aborted due to exception
hudson.AbortException: No test report files were found. Configuration error?
at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:116)
at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:92)
at hudson.FilePath.act(FilePath.java:989)
at hudson.FilePath.act(FilePath.java:967)
at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:89)
at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:121)
at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:138)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:74)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:734)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:683)
at hudson.model.Run.execute(Run.java:1743)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
Finished: FAILURE
To install run .war file, do I need to uninstall the Jenkins as a Windows service first?
Hi Shahir,
no I don’t think you need to completely uninstall the Jenkins service but I would stop it to prevent it from interfering with the .war version of Jenkins.
Are the JUnit XML reports in the location where you told Jenkins to find them?
Hi Bas,
Thanks for your quick reply.
Under ‘Configure’ > ‘Post buld actions’ > ‘Publish JUnit test result report’ > ‘Test report XMLs’ field, I put the path as follows: **/junit/TEST-*.xml
Which means it will take the report from the Jenkins job folder (eg: C:\Program Files (x86)\Jenkins\jobs\SeleniumTest\junit), am I right?
During the Ant build file creation via Eclipse, I specified the output path to the folder above.
But it still fail to load the report. Do I missed something here?
Thanks
That looks pretty good to me. If you go to the Jenkins job folder, are there any results files there?
Yes..the files are there
Another question.
When the selenium script was triggered via Jenkins, why can’t I see the web browser being launched? Does it mean it will always run on ‘silent’ mode? Is there any setting to enable us to see the test being run?
I assume you’re not using the HtmlUnitDriver or the PhantomJSDriver?
It could have something to do with the Jenkins service (assuming you’re running it as a service under Windows) not having the appropriate permissions to open a new window in the same session. In my case the only solution was to use the Jenkins .war instead of installing it as a service, but I don’t know whether that’s the issue on your side.
Hi Bas,
I managed to get the Junit report loaded in Jenkins. Apparently, the path to publish the junit report (during ant build.xml creation) was wrong (C:\Program Files (x86)\Jenkins\jobs\SeleniumTest\junit).
So, I corrected the file path to (C:\Program Files (x86)\Jenkins\jobs\SeleniumTest\workspace\junit) and run the build in Jenkins. Then only Jenkins managed to grab the Junit report and displayed it in web browser.
Such a noob mistake!!
Wasted a few hours googling around for solution..LOL
Glad to see you found the problem!
Hi,
What if I want to run the Selenium script hosted on a Jenkins slave? I have configured the slave and get it connected.
How do I configure Jenkins so that it will run the build.xml that is on the slave machine and not my own machine?
Hi Shahir,
you could for instance give your slave machine a specific label (under Manage Nodes in the Jenkins configuration) and in your job configuration specify that label under ‘Restrict where this job can be run’.
Hi ,
I’m trying to integrate Selenium with junit in Jenkins .Locally my test scripts are running fine with ant build.xml and also generating html results reports.
Whereas when ran in Jenkins , test scripts are passing but html reports are not getting generated.Could you please help me sort this issue?
Thanks
Hi Adit, could you double check your Jenkins job workspace to see whether the report is there yes or no? It could be that the report is generated but not picked up by Jenkins. If that’s all OK, check whether there are any absolute file paths in your build.xml. Those mess up things sometimes!
Hi Bas, i am newbie to the Jenkins. Hope that you can assist me on the Jenkins configuration. And may i know, do i need to configure path in this file TEST-com.ontestautomation.selenium.ci.SeleniumCITest.xml?
And how to solve below issues:
build-project:
[echo] seleniumCI: /var/lib/jenkins/workspace/seleniumCI/build.xml
build:
BUILD SUCCESSFUL
Total time: 0 seconds
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did tests run?
For example, /var/lib/jenkins/workspace/seleniumCI/junit/TEST-com.ontestautomation.selenium.ci.SeleniumCITest.xml is 1 yr 6 mo old
That XML file is the results file, no need to alter that. Judging from your error message either JUnit does not write the report (which I don’t think is very likely) or Jenkins is looking in the wrong folder for the XML test results. Are you able to locate the correct JUnit test result file by hand in your Jenkins job workspace folder?
thanks for your prompt response to this matter.
JUnit test result file is able to access by hand which is stored in the server. But it is still getting error.
Under Post-build Actions>Publish JUnit test result report>est report XMLs I fill it as **/junit/TEST-*.xml
And is the result file stored in that location? I.e., in the junit sub directory for your job, with the file name matching TEST-*.xml?
i copied file name exactly as per what u did sir, which is TEST-com.ontestautomation.selenium.ci.SeleniumCITest.xml.
So under post-build action need to put as:
a. **/reports/TEST-*.xml or
b. target/***/TEST-*.xml or
c. **/TEST-*.xml
please advice
That depends on where your test execution step creates the report. Is that in a subfolder of your job workspace? If not, then c should work. If it goes into a subfolder reports, then a should be correct.
ic understood..
Meanwhile, from console output, what does it means by Test reports were found but none of them are new.. how to clear the old reports?
It means that the reports that were found have a timestamp previous to the latest test run.
Hi ,
I’m trying to integrate Selenium with junit in Jenkins .Locally my test scripts are running fine with ant build.xml and also generating html results reports and the JUnit scenarios are passed.
Whereas running through Jenkins, the Build is getting successful but the JUnit scenarios are getting failed.
Please find below the Error messages:
Scenario 1:
—————————-
@RegressionTest Scenario Outline: Login with Invalid Credentials
org.openqa.selenium.WebDriverException: Failed to navigate to https://www.getlocalhealthplans.com/. This usually means that a call to the COM method IWebBrowser2::Navigate2() failed. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 15 milliseconds
Build info: version: ‘2.45.0’, revision: ‘5017cb8’, time: ‘2015-02-26 23:59:50’
System info: host: ‘XXXXXXXXX’, ip: ‘XXXXXXXXXX’, os.name: ‘Windows 7’, os.arch: ‘x86’, os.version: ‘6.1’, java.version: ‘1.7.0_15’
Session ID: 1e26ddc0-5be1-427e-93ed-1a81bfc7255d
Driver info: org.openqa.selenium.ie.InternetExplorerDriver
Capabilities [{platform=WINDOWS, javascriptEnabled=true, elementScrollBehavior=0, ignoreZoomSetting=false, enablePersistentHover=true, ie.ensureCleanSession=false, browserName=internet explorer, enableElementCacheCleanup=true, unexpectedAlertBehaviour=dismiss, version=11, ie.usePerProcessProxy=false, ignoreProtectedModeSettings=false, cssSelectorsEnabled=true, requireWindowFocus=false, initialBrowserUrl=http://localhost:39436/, handlesAlerts=true, ie.forceCreateProcessApi=false, nativeEvents=true, browserAttachTimeout=0, ie.browserCommandLineSwitches=, takesScreenshot=true}]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:204)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:156)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:599)
at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:304)
at org.openqa.selenium.remote.RemoteWebDriver$RemoteNavigation.to(RemoteWebDriver.java:854)
at integrationTestClass.Cucu_Steps_Code.loginScreen(Cucu_Steps_Code.java:40)
at ✽.Given User is on Login Page(FeatureFile/TestFeature1.feature:5)
Scenario 2:
————————
org.openqa.selenium.UnhandledAlertException: Modal dialog present:
Build info: version: ‘2.45.0’, revision: ‘5017cb8’, time: ‘2015-02-26 23:59:50’
System info: host: ‘XXXXXXXXXX’, ip: XXXXXXXX’, os.name: ‘Windows 7’, os.arch: ‘x86’, os.version: ‘6.1’, java.version: ‘1.7.0_15’
*** Element info: {Using=xpath, value=/html/body/div[1]/div/ul/li[1]/a}
Session ID: 54c7d2ea-3da4-4171-b8df-d0ddc4ea060f
Driver info: org.openqa.selenium.ie.InternetExplorerDriver
Capabilities [{platform=WINDOWS, javascriptEnabled=true, elementScrollBehavior=0, ignoreZoomSetting=false, enablePersistentHover=true, ie.ensureCleanSession=false, browserName=internet explorer, enableElementCacheCleanup=true, unexpectedAlertBehaviour=dismiss, version=11, ie.usePerProcessProxy=false, ignoreProtectedModeSettings=false, cssSelectorsEnabled=true, requireWindowFocus=false, initialBrowserUrl=http://localhost:8979/, handlesAlerts=true, ie.forceCreateProcessApi=false, nativeEvents=true, browserAttachTimeout=0, ie.browserCommandLineSwitches=, takesScreenshot=true}]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:204)
at org.openqa.selenium.remote.ErrorHandler.createUnhandledAlertException(ErrorHandler.java:185)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:152)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:599)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:352)
at org.openqa.selenium.remote.RemoteWebDriver.findElementByXPath(RemoteWebDriver.java:449)
at org.openqa.selenium.By$ByXPath.findElement(By.java:357)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:344)
at org.openqa.selenium.support.ui.ExpectedConditions.findElement(ExpectedConditions.java:730)
at org.openqa.selenium.support.ui.ExpectedConditions.access$0(ExpectedConditions.java:728)
at org.openqa.selenium.support.ui.ExpectedConditions$4.apply(ExpectedConditions.java:130)
at org.openqa.selenium.support.ui.ExpectedConditions$4.apply(ExpectedConditions.java:1)
at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:208)
at integrationTestClass.Cucu_Steps_Code.isElementLoaded(Cucu_Steps_Code.java:563)
at integrationTestClass.Cucu_Steps_Code.login(Cucu_Steps_Code.java:49)
at ✽.When User enters username as “nguyenq@aetna.com” and password as “Justin96″(FeatureFile/TestFeature1.feature:15)
Hi Sam,
Are you running this on the same machine? Could it be that the browser is asking for proxy settings? This because of the UnhandledAlertException..
Hi Bas,
Thanks for your reply. We are running the code in the same machine.
The issue is resolved as of now. We have modified the code as mentioned below:
try {
click(myButton);
} catch (UnhandledAlertException f) {
try {
Alert alert = driver.switchTo().alert();
String alertText = alert.getText();
System.out.println(“Alert data: ” + alertText);
alert.accept();
} catch (NoAlertPresentException e) {
e.printStackTrace();
}
}
Excellent! Have you been able to find out what caused the alert to appear in the first place? Always better to prevent..
No, not yet.. we are still debugging … 🙁
Hi Bas & Sam,
I am facing the scenario 1 as mentioned above. It occurs only when we trigger the build in jenkins. Could you please suggest some solution?
Hi,
Thanks for this wonderful post I have similar problem as others have mentioned.. I am trying to publish junit report and this is the location of my xml file(file name -cucumber.xml) within my Jenkins job –
:C:\Jenkins\jobs\Project_Name\workspace\target\Junit
And I specify in Jekins as follows: **/*cumber.xml
When I run the test I get the following error:
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did tests run?
For example, C:\Jenkins\jobs\Project_Name\workspace\target\Junit\cucumber.xml is 6 days 10 hr old
What happens if you delete that .xml file and run the test again? Is a new one created? I suspect that the actual report is written to a different location…
You are correct when I deleted the xml file and ran the test the file was not created. The error shown was :
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Finished: FAILURE
That confirms that either Jenkins is looking in the wrong directory for the test results or that the test results are not created at all. Could you do a search on your machine for alternative locations where the XML report might have been created?
Hi,
I searched my machine but I could only find the file from my project class path through eclipse but not within my Jenkins job.There seems to be something wrong with my configuration,any idea why I can’t see the report within Jekins directory in machine?
Thanks
Do you use Ant? What does your build.xml look like?
I am using maven,do you intend to see the pom.xml?
What matters is where you define the location for your JUnit reports. Do you do that in your pom.xml?
It’s specified in my CucumberOption plugin from the Runner class,thus gets created in target/Junit/testresult.xml
But my worry is when I deleted it from Jenkins directory,it wasn’t created when I rerun the test as I mentioned earlier.
Yes I got that, my assumption is that is WAS recreated, just not at the location where Jenkins is supposed to pick it up. It’s a bit hard to debug from a distance though 😉
Hi,
Thanks for your help.I will try and run the build by getting the code from the repository and see if it makes any difference because the build running on my VM is getting the code from the repository and I don’t have any problem with the Junit test report getting published.
Excellent. Please let me know if you were able to figure it out!
I resolved the issue by polling from SCM rather than from my local machine.
I have another issue, some tests that pass in my local machine are failing in Jenkins what I mean is that if i run my test from my local machine and Jenkins respectively I have tests that will fail in Jenkins but pass in my local, some pass in both as well.
Why do they fail? What’s the error message?
Ok,basically what happens is this.Not that the tests are not passing at all.Say I run 500 tests,eclipse might have just 3 failures due to synchronisation issue but Jenkins might have 100 failures and 100 tests will be skipped .And the error is usually element not found or element is removed from cache or not clickable but if I compare the result with eclipse most of the tests that fail in Jenkins would have passed in eclipse.
Is this clear a bit?
So if you run your tests locally they run fine but from a build server they don’t? Looking at your description of the errors you might want to improve your error handling a bit. I’ve written a post on wrapper methods some time ago, maybe that one can give you some pointers?
Thanks but my worry is why do the same tests pass in my local?isn’t it something to do with Jenkins server or some configuration.Some people say if Jenkins is being run as a window service that it may happen but I am not running Jenkins as a window service.
No, I don’t think it is caused by Jenkins running as a Windows service or not, because that would either result in no failed tests or all tests failing. The fact that SOME tests fail indicates that it is very likely a timing thing, something that you can try to handle by using better exception handling, for instance using wrapper methods.
Can the same approach be used to integrate selenium with Hybris?
I am actually accessing a localhost url, which calls for starting the hybris server first.
How shall I work around with that?
Hey Kaveri,
why not? You can simply start the Hybris server with a batch command from Jenkins before you run your tests.
Hi Bas,
My Selenium Junits works fine when i run locally. But, when i run them on hudson, i always get run failed. None of them are passed.
Can you please help me ?
What’s the error message you’re seeing? I can’t guess what’s different in your Hudson setup vs. your local machine from here…
Hi,
When i run multiple instances parallely in Jenkins. some of the testcases are getting failed. So is there any solution to increase the performance ??
I think it would be better to see why they are failing in the first place. Are there tests that have race conditions, do they accidentally use and mutate the same test data or do they depend on one another?