databene

 
  • Increase font size
  • Default font size
  • Decrease font size

Feed4JUnit

[--> Srpsko-Hrvatski]

Feed4JUnit makes it easy to write parameterized tests for the JUnit framework and feed them with predefined or random test data. It enables you to improve your testing easily:
  • Reading test case data from CSV or Excel files defined by a business analyst and reporting test success within the build/unit test infrastructure
  • Getting test data from databases or custom data sources
  • Easily performing smoke tests with random but valid data can strongly improve code coverage and exhibit bugs caused by very special data constellations
  • Defining equivalence class tests easily, which is a powerful methodology of software testers, but little known among developers
Configuration is easy to learn and to maintain and based on Java annotations. They can be used to import data from files (CSV, Excel) or to generate random data that matches data constraints declared with execptions from the "Bean Validation" JSR 303 and Java 7 or from Benerator. By connecting to Benerator, the world's leading open source test data generator, you can configure generation of arbitrary complex valid and invalid data sets.

 

Introduction

Test Basics

Feed4JUnit use a special Runner class called Feeder to support and feed parameterized unit tests:

@RunWith(Feeder.class)
public class LoginTest {

    @Test
    public void testLogin(String name, String password) {
        System.out.println("name:" + name + " password:" + password);
    }

}

For a smoke tests that's enough to start! Run this test with your favorite JUnit environment and Feed4JUnit will use Benerator's power to automatically find interesting values for testing (applying ideas from border value testing) and combine them (trying to cover all resulting equivalence partitions). Eclipse will then show a report like this, listing each single test method calls with its parameters:

TODO

As you can see, Feed4JUnit tries to feed each string parameter with null, an empty string, a one-character-string, a long string (1000 chars) and a medium sized string (500 chars), checks borders of the character set (A and Z), a value in between (N) and checks each possible combination of the different parameters.

Of course you can use annotations to restrict the range of generated data.

 

Reading test data from a file

Data sources are easily configured using the @Source annotation (org.databene.benerator.anno.Source):

@RunWith(Feeder.class)
public class LoginTest {

    @Test
    @Source("userlogin.csv")
    public void testLogin(String name, String password) {
          System.out.println("name:" + name + " password:" + password);
    }
   

Running the test, you will notice, that Feed4JUnit iterates through the CSV file and performs one test method call per data row.

Feed4JUnit currently supports the following file formats:

  • CSV files
  • Excel(TM) documents 

 

Retrieving test data from a database

Databases can be declared and configured with a @Database annotation. When applying the annotation to a class, the defined database is available to all test methods. When annotating a method, the database is available only in this method. A basic configuration can be performed by specifying the typical JDBC connection settings: url, driver, user, password. The database declaration must specify an id, by which the database can be referred to as @Source of a method. The @Source annotation must refer to the database id and define a SQL query as selector. The number of query result columns must match the number of method parameters:

@RunWith(Feeder.class)
@Database(id = "db", url = "jdbc:hsqldb:hsql://localhost:9001/f4jdb",
        driver = "org.hsqldb.jdbcDriver", user = "me", password = "secret")
public class DatabaseTest { 
    static DBSystem db;
   
    @Test
    @Source(id = "db", selector = "select id, name from dbt_person")
    public void test(int id, String name) {
        System.out.println(id + ", " + name);
    }
   
}

Alternatively to the explicit database connection configuration, you can place database environment configurations in your user directory and refer to them in the test:

@Database(id = "db", environment = "f4jdb")

Read more about environment files in the DB Sanity environment files documentation.

 

Defining custom data sources

If you wish to retrieve data from other data source or file types, you can program a connector component, that inherits from the class Feed4JUnitGenerator and provides data:

public static class MyGenerator extends UnsafeMethodParamsGenerator {

        public Object[] generate() {
            return new Object[] { 1, 2, 3 };
        }

}

The Object array returned by the generate() method is used as parameter list for the invocation of the test method. If the data available from a source is depleted (eg. the end of file is reached), the generator class returns null.

The example class MyGenerator returns { 1, 2, 3 } on each invocation, thus it would be called endlessly, unless you annotate the related test methods with an @InvocationCount for limiting the data volume.

A custom generator class is declared by a @Bean annotation with an id and instructions how to instantiate and initialize the bean object. A @Source annotation at a method referes to its id:

@RunWith(Feeder.class)
@Bean(id = "mygen", type = MyGenerator.class)
public class BeanSourceTest {
   
    @Test
    @Source("mygen")
    @InvocationCount(5)
    public void testArrayIterable(int n1, int n2, int n3) {
        System.out.println(n1 + ", " + n2 + ", " + n3);
    }
   
}

In this case, the type specification tell Feed4JUnit to instantiate an object of the class MyGenerator by its default constructor.

Alternatively, you can make use of Benerator's full feature set regarding bean instantiation, eg. calling a constructor:

@Bean(id = "mygen", spec = "new MyOtherGenerator(1, 'test')")

or using a bean property contructor: 

@Bean(id = "mygen", spec = "new MyOtherGenerator{ property1=1, property2 = 'test'}")

or the more bean-like approach:

@Bean(id = "mygen", type = "MyOtherGenerator", properties = {
    @Property(name = "property1", value = "1"),
    @Property(name = "property2", value="test")
})

 

Generating constrained data dynamically

Feed4JUnit supports annotations defined in JSR 303, Java 7 and Benerator 0.7 for generating random data that matches constraints.

As an example see the @Pattern annotation as an example (javax.validation.constraints.Pattern):

@RunWith(Feeder.class)
public class RegexTest {

    @Test
    public void testSmoke(@Pattern(regexp = "[A-Z][a-z]{3,8}") String name) {
        System.out.println("name:" + name);
    }
   
}

 

Annotation reference

Annotations marked bold are new in version 1.0:

AnnotationJava PackageApplicability
Description
@AssertFalsejavax.validation.constraintsparameter
Requires that a boolean parameter is false
@AssertTruejavax.validation.constraintsparameter
Requires that a booöean parameter is true
@DecimalMinjavax.validation.constraintsparameter
Requires that a number parameter is greater than or equals a minimum value
@DecimalMax
javax.validation.constraintsparameter
Requires that a number parameter is less than or equals a maximum value 
@Future
javax.validation.constraintsparameter
Requires that a Date parameter is in the future
@Min
javax.validation.constraintsparameter
Requires that an integral number parameter is greater than or equals a minimum value
@Max
javax.validation.constraintsparameter
Requires that an integral number parameter is less than or equals a maximum value
@NotNull
javax.validation.constraintsparameter
Requires that a parameter is not null
@Null
javax.validation.constraintsparameter
Requires that a parameter is null
@Past
javax.validation.constraints
parameter
Requires that a Date parameter is in the past
@Pattern
javax.validation.constraints
parameter
Requires that a String parameter matches a regular expression
@InvocationCount
org.databene.benerator.anno
method
Limits the number of invocations to a test method
@Source
org.databene.benerator.anno
method, parameter
Specifies a source file from which to read test data (e.g. CSV or Excel(TM) file)
@Offset
org.databene.benerator.anno method,
parameter
Makes Feed4Junit skip the first n data sets that are imported or generated
@Distribution
org.databene.benerator.anno
parameter
Specifies a distribution to use for a number parameter
@Granularity
org.databene.benerator.anno
parameter
Specifies the granularity of a number parameter (corresponds to Benerator's 'precision')
@Nullquota
org.databene.benerator.anno
parameter
Specifies the quota of null values to generate for a parameter
@Values
org.databene.benerator.anno
parameter
Specifies a comma-separated list of all supported values
@Last
org.databene.benerator.anno
parameter

Used for summary and cleanup functionality necessary after the last call to a test method: The last parameter of a mathod can be annotated with @Last and be made booloean. It will then receive a true value on the last invocation, otherwise false.

@Generator
org.databene.benerator.anno
method, parameter
Specifies a simple type generator for a parameter or an array generator for a complete parameter set
@Database
org.databene.benerator.anno

class,
method

Defines a database to be used for data retrieval
@Beanorg.databene.benerator.annoclass,
method
Defines a custom data generator to retrieve data from
@Equivalence
org.databene.benerator.anno
class,
method

Advises Feed4JUnit to use the EquivalenceGeneratorFactory for all related test methods. It creates relatively small numbers of mean tests using mechanisms from border value and partition testing.
@Coverage
org.databene.benerator.anno
class,
method

Advises Feed4JUnit to use the CoverageGeneratorFactory for all related test methods. It runs through all possible values and all possible combinations and produces a large number of tests.
@Stochastic org.databene.benerator.anno
class,
method

Advises Feed4JUnit to use the StochasticGeneratorFactory for all related test methods. It creates gentle data randomly and provides for an unlimited number of tests.

 

Testing Methodologies

Equivalence Class Testing

Equivalence class tests are a powerful methodology of software testers, but little known among developers. Feed4JUnit makes it easy to use for developers.

Fundamental ideas of the approach are 

  • Defining 'classes' or groups of values which are processed similarly in the code under test (e.g. negative numbers, zeros, positive numbers)
  • Testing at least one instance of each test class (e.g. -6, 0, 42)
  • Testing border values (e.g. MIN_INT, -1, 0, 1, MAX_INT)
  • If the test has several parameters, check each valid combination of the values defined above (MIN_INT, -6, -1, 0, 1, 42, MAX_INT) for each parameter.

If all parameters have the same equivalence classes, our data set would lead to the following number of tests:

Number of
parameters

Number of
tests
1
7
249
3
343

You would not want to code 49 or even 343 test setups by hand, would you? You would not cover all, but rely on a more or less arbitrary choice. Feed4JUnit makes it easy to check all relevant cases by automatically forming a cartesian product of the single parameters' values and use each of them.

 

Smoke Testing

Performing smoke tests with random but valid data can strongly improve code coverage and exhibit bugs from very special data constellations. Even if you cannot simply predict each result of random data, you can check result constraints or at least check for runtime exceptions.

In FeedJUnit, you can use the @Stoachstic annotation to generate random data or the @Coverage annotation to first generate border values and then all values between. Be aware that you need to restrict the InvocationCount in many cases (unless you want to test billions of calls):

@RunWith(Feeder.class)
public class AddTest {

    @Test
    @Coverage
    @InvocationCount(100)
    public void testAdd(int param1, int param2) {
        try {
            int result = MyUtil.add(param1, param2);
        } catch (Exception e) {
            // accept application exceptions, fail on runtime exceptions
            // like NullPointerException

           
if (e instanceof RuntimeException)
                throw e;

        } 
    }

}

 

Analyst Driven Tests

Analysts like Excel. Developers do so too, but mostly have to manually transfer data from test cases defined in tables into test code. If requirements change, it is a tedious task to rescan test code and rematch it with table data. Much easier and more maintainable is to automatically read the table from the Excel file and feed it to the test code. The data file may even reside on a web page oran FTP server and dynamically retrieved from there when executing the test. So n analyst does not even have to cope with versioning systems to interoperate with development.

@RunWith(Feeder.class)
public class AddTest {

    @Test
    @Source("http://buildserv.mycompany.com/wiki/myproject/tests/add.xls")
    public void testAdd(int param1, int param2, int expectedResult) {
        int result = MyUtil.add(param1, param2);
        assert result == expectedResult;
    }

}

You can retrieve files from an FTP server with a special URL syntax: @Source("ftp://user:password@server/path/file.txt")

 

Using Feed4Junit in your project

Using Feed4JUnit in an IDE

For running the test in a project in your IDE (e.g. Eclipse), download the feed4junit distribution and add all files contained in its lib folder to your project's class path.

Write a Feed4JUnit test and execute it.

For each test invocation you will then find a dedicated entry with the used parameters in your test protocol. Eclipsewill display a JUnit view similar to this:

 

Using Feed4JUnit in a Maven build

Lay out your project and its pom as you are used to, then add a dependency to the newest version of Feed4JUnit:

    <dependency>
        <groupId>org.databene</groupId>
        <artifactId>feed4junit</artifactId>
        <version>1.0.1</version>
    </dependency>

After executing the tests e.g. using 'mvn test', you can find two files for each test class in the folder target/surefire-reports/. The files TEST-<fully_qualified_classname>.xml provide the parameter data of each test method invocation, e.g.

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite failures="0" time="0.01" errors="0" skipped="0" tests="10" name="LoginDbTest">
...
<testcase time="0.001" classname="LoginDbTest" name="testLoginDb[Alice, 1234]"/>
<testcase time="0" classname="LoginDbTest" name="testLoginDb[Bob, 0000]"/>
</testsuite>

When running 'mvn site', Maven will generate a test report with one line per test invocation, displaying the used parameters like this:

 

Special CSV and Excel(TM) Sheet Features

Feed4JUnit supports the import of CSV files and Excel(TM) Sheets, formulas in Excel Sheets are resolved. In the following chapters, the term data file represents CSV files as well as Excel files.

Data file per method

When annotating a test method with a @Source that refers to a data file, each column is mapped to a method parameter sequentially, independent of the column name and the parameter name:

    @Test
    @Source("users.ent.csv")
    public void testMethodSource(String name, int age) {
        ...
    }

The corresponding data file is expected to have a header row and an arbitrary number of rows. Each row represents one data set with which the test method is called. If the file users.ent.csv has the following content:

name,age
Alice,23
Bob,34

the test method is called twice: First with the parameters (Alice, 23), then with the parameters (Bob, 34).

 

CSV format

Feed4JUnit supports CSV as specified in RFC 4180. Thus, cells may be double-quoted, e.g. to include the separator character as plain character. As an example, the line

Alice,"Alice","Bob,Charly"

is parsed into a data set of three string values, which the first two entries contain the string Alice and the third one the string Bob,Charly. Note that each one is provided without quotes.

For differing between empty strings and null values, empty double quotes are mapped to a string, an empty cell is mapped to a null value:

Alice,,""

is mapped to a data set of three values, the first one is the string Alice, the second one null, the third one an empty string.

 

Excel(TM) format

Excel sheets are imported using the Apache POI library which provides any numerical value as a floating point value. So be prepared that some fractional numbers are not parsed exactly as you expect them to be.

Empty cells are mapped to null values, The default notation for an empty string in Excel is one single-quote character in a cell. Unfortunately this is not displayed in the sheet view, so the user needs to enter each cell in order to tell if its value represents null or an empty string. If the difference between null and empty strings matters for you, Excel sheets would become much more intuitive if each visually empty cell is indeed empty and mapped to a null value and each cell that represents an empty string, displays a marker text. This approach can be used by Feed4JUnit's emptyMarker.

The user can chose a custom marker text to represent empty strings, e.g. <empty>, and use it in an Excel sheet:

Then the emptyMarker needs to be declared in Feed4JUnit's @Source annotation:

    @Test
    @Source(uri = "values.ent.xls", emptyMarker = "<empty>")
    public void testMethodSource(String col1, String col2) {
        ...
    }

Finally, when running the tests, the values which match the emptyMarker are replaced with empty strings. The example yields the data sets:

1. Alice and an empty string
2. Bob an a null value

 

Column-based data

For tests with extraordinarily long parameter lists and few test cases to execute, a column-based data format might be more convenient for the user. In this case, the first column is expected to be the header column and each further column represents a parameter set for a test method invocation. The data of the example above would be represented like this:

name,Alice,Bob
age,23,34

and the @Source annotation needs a flag rowBased = false:

    @Test
    @Source(uri = "user-columns.ent.csv", rowBased = false)
    public void testMethodColumnSource(String name, int age) {
        ...
    }

 

JavaBean parameters

Data for a JavaBean parameter can be read from a data file. Suppose you have a JavaBean class Country with default constructor and the properties 'isoCode' and 'name':

public class Country {
   
    private String isoCode;
    private String name;

    public String getIsoCode() {
        return isoCode;
    }

    public void setIsoCode(String isoCode) {
        this.isoCode = isoCode;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
   
}

Then you can annotate a method's JavaBean-type-parameter with a @Source annotation that points to the file:

    @Test
    public void testCsvBean(@Source("countries.ent.csv") Country country) {
        ...
    }

In this case, the column names of the data file matter and are mapped to the bean's properties of same name.

isoCode,name
DE,Germany
US,United States

 

Reading JavaBean graphs from a data file

Bean graphs can also be imported from a single data file. Suppose you have a 'User' JavaBean which references the 'Country' JavaBean:

public class User {
   
    private String name;
    private int age;
    private Country country;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public int getAge() {
        return age;
    }

    public void setAge(int age) {
        this.age = age;
    }
   
    public Country getCountry() {
        return country;
    }
   
    public void setCountry(Country country) {
        this.country = country;
    }

}

Feed4JUnit recognizes that the User's 'country' property is a JavaBean and when importing file data for a User

    @Test
    public void testCsvNestedBean(@Source("users2.ent.csv") User user) {
        ...
    }

it maps the data appropriately based on recursive property naming:

name,age,country.isoCode,country.name
Alice,23,DE,Germany
Bob,34,US,USA

 

Locating data files

Feed4JUnit uses the following default strategy for looking up file names specified in a @Source annotation's uri:

  1. absolute (if it is an absolute path like C:\tests\my.csv or /tests/my.csv)
  2. class path
  3. relative to the working directory

For alternative lookup strategies, a service provider interface is supported: org.databene.benerator.anno.PathResolver.

By default, Feed4JUnit uses the org.databene.benerator.anno.DefaultPathResolver which employs the behaviour described above.

An alternative strategy is provided by org.databene.benerator.anno.RelativePathResolver: It uses a configurable base path, adds the package name of the test to run (as path components) and finally adds the resource path specified as URI. So, when using a base path C:\tests, an annotation @Source("p1/user.ent.csv") used in test class com.my.UserTests is mapped to the file path C:\tests\com\my\p1\user.ent.csv.

If you need alternative strategies, you can write a custom implementation of the PathResolver interface.

For choosing a specific path resolver, put a file feed4junit.properties into your project (or working) directory and specify a pathResolver:

pathResolver=new org.databene.benerator.anno.RelativePathResolver('C:\\test')

The example shows how to configure the RelativePathResolver using the base path C:\test.

The pathResover expression is interpreted using DatabeneScript which supports most Java expression types. A main difference is, that strings literals use single quotes as shown in the example. Also note that string values are interpreted Java-like, so the Windows path separator \ is represented by its Java escape character. 

 

Further Information

License

Feed4JUnit is Open Source, released under GNU Public License, GPL 2.0. Note however that it requires Benerator which has a different license, a GPL 2.0 with exceptions.

 

Requirements

  • Feed4JUnit requires Java 6 or newer
  • Using JUnit as the underlying framework for defining a test. If you want to use TestNG, you can do so with Feed4TestNG: It extends TestNG with support for parameterized tests and provides the same features as Feed4JUnit with a slightly different syntax.

 

Limitations

  • Since JUnit does extensive logging of each test invocation, Feed4Junit is not suitable for microbenchmarking or load testing. 
  • Performing automated equivalence class tests is planned for Feed4JUnit, but the underlying data generation infrastructure does not yet support it. I will implement that in future releases.

 

Help & Suggestions

If you are stuck, found a bug, have ideas for Feed4JUnit or want to help, visit the forum.