databene

 
  • Increase font size
  • Default font size
  • Decrease font size

Feed4TestNG

Feed4TestNG makes it easy to write parameterized tests for the TestNG framework and feed them with predefined or random test data. It enables you to improve your testing easily:

  • Reading test case data from CSV or Excel files defined by a business analyst and reporting test success within the build/unit test infrastructure.
  • Easily performing smoke tests with random but valid data can strongly improve code coverage and exhibit bugs caused by very special data constellations.
  • Defining equivalence class tests easily, which is a powerful methodology of software testers, but little known among developers
Configuration is easy to learn and to maintain and based on Java annotations. They can be used to import data from files (CSV, Excel) or to generate random data that matches data constraints declared with execptions from the "Bean Validation" JSR 303 and Java 7 or from Benerator. By connecting to Benerator, the leading test data generator, you can configure generation of arbitrary complex valid and invalid data sets.

 

Introduction

Test Basics

Feed4TestNG tests must extend the class org.databene.feed4testng.FeedTest and use the dataProvider called "feeder":

public class LoginTest extends FeedTest {

    @Test(dataProvider = "feeder")
    public void testLogin(String name, String password) {
        System.out.println("name:" + name + " password:" + password);
    }
   
}

For a smoke tests that's enough to start! Run this test with your favorite JUnit environment and Feed4JUnit will use Benerator's power to automatically find interesting values for testing (applying ideas from border value testing) and combine them (trying to cover all resulting equivalence partitions). When executing the tests with TestNG, you will see, that Feed4TestNG automatically generates a mix of critical and simple values for test invocation. For a string, these are null, an empty string, a one-character-string, a long string (1000 chars) and a medium sized string (500 chars), checks borders of the character set (A and Z), a value in between (N) and checks each possible combination of the different parameters.

Of course you can use annotations to restrict the range and characteristics of generated data.

 

 

Reading test data from a file

Data sources are easily configured using the @Source annotation (org.databene.benerator.anno.Source):

public class LoginTest extends FeedTest {

    @Test(dataProvider = "feeder")
    @Source("userlogin.csv")
    public void testLogin(String name, String password) {
        System.out.println("name:" + name + " password:" + password);
    }
   
} 

Running the test, you will notice, that Feed4TestNG iterates through the CSV file and performs one test method call per data row.

Feed4TestNG currently supports the following file formats:

  • CSV files
  • Excel(TM) sheet 

Retrieving test data from a database

Databases can be declared and configured with a @Database annotation. When applying the annotation to a class, the defined database is available to all test methods. When annotating a method, the database is available only in this method. A basic configuration can be performed by specifying the typical JDBC connection settings: url, driver, user, password. The database declaration must specify an id, by which the database can be referred to as @Source of a method. The @Source annotation must refer to the database id and define a SQL query as selector. The number of query result columns must match the number of method parameters:

@Database(id = "db", url = "jdbc:hsqldb:hsql://localhost:9001/f4jdb",
        driver = "org.hsqldb.jdbcDriver", user = "me", password = "secret")
public class DatabaseTest {
   
    static DBSystem db;
   
    @Test(dataProvider = "feeder")
    @Source(id = "db", selector = "select id, name from dbt_person")
    public void test(int id, String name) {
        System.out.println(id + ", " + name);
    }
   
} 

Alternatively to the explicit database connection configuration, you can place database environment configurations in your user directory and refer to them in the test:

@Database(id = "db", environment = "f4jdb") 

Read more about environment files in the DB Sanity environment files documentation.

 

Defining custom data sources

If you wish to retrieve data from other data source or file types, you can program a connector component, that inherits from the class Feed4TestNGGenerator and provides data:

public static class MyGenerator extends UnsafeMethodParamsGenerator {
    public Object[] generate() {
        return new Object[] { 1, 2, 3 };
    }
} 

The Object array returned by the generate() method is used as parameter list for the invocation of the test method. If the data available from a source is depleted (eg. the end of file is reached), the generator class returns null.

The example class MyGenerator returns { 1, 2, 3 } on each invocation, thus it would be called endlessly, unless you annotate the related test methods with an @InvocationCount for limiting the data volume.

A custom generator class is declared by a @Bean annotation with an id and instructions how to instantiate and initialize the bean object. A @Source annotation at a method referes to its id:

@RunWith(Feeder.class)
@Bean(id = "mygen", type = MyGenerator.class)
public class BeanSourceTest {
   
    @Test(dataProvider = "feeder")
    @Source("mygen")
    @InvocationCount(5)
    public void testArrayIterable(int n1, int n2, int n3) {
        System.out.println(n1 + ", " + n2 + ", " + n3);
    }
   
} 

In this case, the type specification tell Feed4TestNG to instantiate an object of the class MyGenerator by its default constructor.

Alternatively, you can make use of Benerator's full feature set regarding bean instantiation, eg. calling a constructor:

@Bean(id = "mygen", spec = "new MyOtherGenerator(1, 'test')") 

or using a bean property contructor: 

@Bean(id = "mygen", spec = "new MyOtherGenerator{ property1=1, property2 = 'test'}") 

or the more bean-like approach:

@Bean(id = "mygen", type = "MyOtherGenerator", properties = {
    @Property(name = "property1", value = "1"),
    @Property(name = "property2", value="test")
}) 

 

Generating constrained data dynamically

Feed4TestNG supports the annotations defined in JSR 303, Java 7 and Benerator for generating random data that match constraints.

See the @Pattern annotation as an example (javax.validation.constraints.Pattern):

public class SmokeTest extends FeedTest {

    @Test(dataProvider = "feeder")
    public void testSmoke(@Pattern(regex = "[A-Z][a-z]{3,8}") String name) {
        System.out.println("name:" + name);
    }
   
}

The following annotations are supported:

AnnotationJava PackageApplicability
Description
@AssertFalsejavax.validation.constraintsparameter
Requires that a boolean parameter is false
@AssertTruejavax.validation.constraintsparameter
Requires that a booöean parameter is true
@DecimalMinjavax.validation.constraintsparameter
Requires that a number parameter is greater than or equals a minimum value
@DecimalMax
javax.validation.constraintsparameter
Requires that a number parameter is less than or equals a maximum value 
@Future
javax.validation.constraintsparameter
Requires that a Date parameter is in the future
@Min
javax.validation.constraintsparameter
Requires that an integral number parameter is greater than or equals a minimum value
@Max
javax.validation.constraintsparameter
Requires that an integral number parameter is less than or equals a maximum value
@NotNull
javax.validation.constraintsparameter
Requires that a parameter is not null
@Null
javax.validation.constraintsparameter
Requires that a parameter is null
@Past
javax.validation.constraints
parameter
Requires that a Date parameter is in the past
@Pattern
javax.validation.constraints
parameter
Requires that a String parameter matches a regular expression
@InvocationCount
org.databene.benerator.anno
method
Limits the number of invocations to a test method
@Source
org.databene.benerator.anno
method, parameter
Specifies a source file from which to read test data (e.g. CSV or Excel(TM) file)
 @Offsetorg.databene.benerator.anno
method,
parameter
 
@Distribution
org.databene.benerator.anno
parameter
Specifies a distribution to use for a number parameter
@Granularity
org.databene.benerator.anno
parameter
Specifies the granularity of a number parameter (corresponds to Benerator's 'precision')
@Nullquota
org.databene.benerator.anno
parameter
Specifies the quota of null values to generate for a parameter
@Unique
org.databene.benerator.anno
method, parameter If applied to a parameter it specifies that its values must be unique. Applied to a method it specifies that the combination of all parameters must be unique
@Values
org.databene.benerator.anno
parameter
Specifies a comma-separated list of all supported values
@Generator
org.databene.benerator.anno
method, parameter
Specifies a simple type generator for a parameter or an array generator for a complete parameter set
@DescriptorBased
org.databene.benerator.anno
method
Specifies that the parameters of a test method shall be generated as configured in a Benerator descriptor file
@Equivalence
org.databene.benerator.anno
class,
method

Advises Feed4JUnit to use the EquivalenceGeneratorFactory for all related test methods. It creates relatively small numbers of mean tests using mechanisms from border value and partition testing.
@Coverage
org.databene.benerator.anno
class,
method

Advises Feed4JUnit to use the CoverageGeneratorFactory for all related test methods. It runs through all possible values and all possible combinations and produces a large number of tests.
@Stochastic 
org.databene.benerator.anno
class,
method

Advises Feed4JUnit to use the StochasticGeneratorFactory for all related test methods. It creates gentle data randomly and provides for an unlimited number of tests. 

 

Testing Methodologies

Equivalence Class Testing

Equivalence class tests are a powerful methodology of software testers, but little known among developers. Feed4TestNG makes it easy to use for developers.

Fundamental ideas of the approach are 

  • Defining 'classes' or groups of values which are processed similarly in the code under test (e.g. negative numbers, zeros, positive numbers)
  • Testing at least one instance of each test class (e.g. -6, 0, 42)
  • Testing border values (e.g. MIN_INT, -1, 0, 1, MAX_INT)
  • If the test has several parameters, check each valid combination of the values defined above (MIN_INT, -6, -1, 0, 1, 42, MAX_INT) for each parameter.

If all parameters have the same equivalence classes, our data set would lead to the following number of tests:

Number of
parameters

Number of
tests
1
7
249
3
343

You would not want to code 49 or even 343 test setups by hand, right? You would not cover all, but rely on a more or less arbitrary choice.

 

Smoke Testing

Performing smoke tests with random but valid data can strongly improve code coverage and exhibit bugs from very special data constellations. Even if you cannot simply predict each result with random data, you can check result constraints or at least check for runtime exceptions:

public class AddTest extends FeedTest {

    @Test(dataProvider = "feeder")
    @Coverage
    public void testAdd(int param1, int param2) {
        try {
            int result = MyUtil.add(param1, param2);
        } catch (Exception e) {
            // accept application exceptions, fail on runtime exceptions
            // like NullPointerException

           
if (e instanceof RuntimeException)
                throw e;

        } 
    }

}

 

Analyst Driven Tests

Analysts like Excel. Developers do so too, but mostly have to manually transfer data from test cases defined in tables into test code. If requirements change, it is a tedious task to rescan test code and rematch it with table data. Much easier and more maintainable is to automatically read the table from the Excel file and feed it to the test code. The data file may even reside on a web page oran FTP server and dynamically retrieved from there when executing the test. So n analyst does not even have to cope with versioning systems to interoperate with development.

public class AddTest extends FeedTest {

    @Test(dataProvider = "feeder")
    @Source("http://buildserv.mycompany.com/wiki/myproject/tests/add.xls")
    public void testAdd(int param1, int param2, int expectedResult) {
        int result = MyUtil.add(param1, param2);
        assert result == expectedResult;
    }

}

You can retrieve files from an FTP server with a special URL syntax: @Source("ftp://user:password@server/file.txt")



Using Feed4TestNG in your project

Using Feed4TestNG in an IDE

For running the test in a project in your IDE (e.g. Eclipse), download the feed4testng distribution and add all files contained in its lib folder to your project's class path.

If you do not know how to make Eclipse run TestNG tests, read this and install the TestNG Eclipse plugin.

Write a Feed4TestNG test and execute it.

For each test invocation you will then find a dedicated entry with the used parameters in your test protocol.

 

Using Feed4TestNG in a Maven build

Lay out your project and its pom as you are used to, then add a dependency to the newest version of Feed4TestNG:

    <dependency>
        <groupId>org.databene</groupId>
        <artifactId>feed4testng</artifactId>
        <version>1.0</version>
    </dependency>

After executing the tests e.g. using 'mvn test', you can find a deidcated entry for each test method invocation along with the used paramers in the file target/surefire-reports/testng-results.xml, e.g.

        <test-method status="PASS" signature="testDescriptor(java.lang.String, java.lang.String)" 
name="testDescriptor" duration-ms="0" started-at="2010-06-04T19:41:37Z"
finished-at="2010-06-04T19:41:37Z">
          <params>
            <param index="0">
              <value>
                <![CDATA[7962]]>
              </value>
            </param>
            <param index="1">
              <value>
                <![CDATA[MORRISTOWN]]>
              </value>
            </param>
          </params>
        </test-method>

When running 'mvn site', Maven will generate a test report with one line per test invocation, but it will not display the used parameters.

 

Further Information

License

Feed4TestNG is Open Source, released under GNU Public License, GPL 2.0. Note however that it requires Benerator which has a different license, a GPL 2.0 with exceptions.

 

Requirements

  • Feed4TestNG requires Java 6 or newer
  • Using TestNG as the underlying framework for defining a test. If you want to use JUnit, you can do so with Feed4JUnit: It extends JUnit with support for parameterized tests and provides the same features as Feed4TestNG with a slightly different syntax.

 

Limitations

  • Feed4TestNG creates and writes extensive execution information to a log file. So Feed4TestNG is not appropriate for feeding micro benchmarks.
  • Performing automated equivalence class tests is planned for Feed4TestNG, but the underlying data generation infrastructure does not yet support it. I will implement that in future releases.

 

Help & Suggestions

If you are stuck, found a bug, have ideas for Feed4TestNG or want to help, visit the forum.