Behaviour of @Before and @After

[nicolas]

The behaviour for @After and @Before with ContiPerfs is very puzling as far as I am concerned. In JUnit (v4.7 and v4.10), @Before methods are called before EACH call to a test method of the class (similar behaviour for @After) :

before()
test1()
after()
before()
test2()
after()

but with ContiPerf, we are supposed to deal with a different logic, as you mentionned in a previous post (viewtopic.php?f=10&t=263#p910) :
before()
test1()
test2()
after()

Is this really the wanted behaviour for ContiPerfs ? Is there a way to execute the sequence of calls of my first example ?

Thank you,

PS : I've seen two other topics around this subject, but not exactly in the aim of my concern.

[Volker Bergmann]

Hi Nicolas,

sorry, this is a misunterstanding and I hope to make it definitely clear now:

Since ContiPerf 2.0.0 the behavior is uniform among JUnit 4.7, 4.8 and 4.9 and is as follows:

Code: Select all

before()
test1()
test1()
test1()
after()

before()
test2()
test2()
test2()
after()

This is the best way to transfer JUnit semantics of the @Before and @After methods to performance testing.

If you want to have one initialization method called exactly once before any test execution and one after the last test, you need to use @BeforeClass and @AfterClass:

Code: Select all

beforeClass()
before()
test1()
test1()
test1()
after()

before()
test2()
test2()
test2()
after()
afterClass()

Regards,

[nicolas]

Ok, Thank you for this reply, I realize that I haven't been clear enough with my explanations (Indeed, I don't want to "have one initialization method called exactly once before any test execution and one after the last test" as you thought in your reply).

I try again : If we consider that semantic of "@before" and "@after" aims to set an initial state and clear the final state for each unit call, I would like to be able to "see" this with contiPerf :

Code: Select all

before()
test1()
after()
before()
test1()
after()

before()
test2()
after()
before()
test2()
after()

At the moment, as contiPerf is designed to link a second test1() call directly to the first test1() call, the concern is that the second call will start based on the final state of the first test1() call, what is, in my opinion, not necessarly expected.

Let's check this case :

Code: Select all

@Before
public void init(){
   //clear the DB
}

@Test
public void test1(){
   //insert 1 row
   // assertTrue that there is 1 row and only 1 row in the table
}

If I use a contiPerf annnotation in order to make 2 invocations for test1(), the second call will fail, right ?

Writing this post, I wonder if I am not totally wrong with my use of contiPerf : maybe It's not a good idea to mix my "business" JUnit tests with contiPerf ? Shall I dedicate classes only for performance tests with contiPerf ? Just let me know your thinkings about this.

Even if it were not a good idea to mix business and performance tests, for the moment I am unable to implement something like this :

Code: Select all

public class LoginTest {

private String login;
private String password;

  @Before
  public void init(){
     //set login and password attributes from a list (randomly)
  }

  @Test
  public void testPerfLogin{
    //call a login method to test, passing the parameters "login" and "password"
  }
}

because only one couple of login/password will be used for the entire performance test.

If my concern is "valid" : I suppose we could imagine to introduce some new contiPerfs annotations, such as "@BeforeEachInvocation" and "@AfterEachInvocation" ?

Thank you !

[Volker Bergmann]

The requested feature does not exit - and this is by purpose: You want to measure the performance of the tested code - right? When executing set-up code, the tested system has to process different types of work, when testing concurrently (and only this really makes sense for most performance testing tasks), the system may exhibit a very different behaviour than when ruuning only with the tested code.

By the way, if accidentally or not, your example exhibits a fatal approach which often leads to drastic overestimations on system performance: Performance testing on an empty database! Measuring the time needed to insert one single row in an empty database is of no use for anything! Doing so will make you report illusionary performance numbers to your boss and cause you serioous problems when production is stalled by 10% of the reported performance. Unfortunately, this is one of the default testing mistakes. Please google for the performance testing anti-pattern 'Datalite' and read about it.

So, always prepare a database with a data volume comparable to production before executing performance tests.

For performance testing a database, it is also important to test on a wide range of database objects, so using always the same user or just a small number of users for login testing (as an example) is likely to make you test your database cache performance instead of the database performance itself. You could write a SQL statement that provides you with the core data needed to perform the test on a write range of objects in a before method and then iterate the data in the test method. For larger amounts of test data, you could make a query that incurs randomness, store the result to a CSV file and then iterate the CSV file in the test method.

There are way more things to regard, but these are some of the fundamentals. BTW: I am selling performance workshops as well and with these I provide test examples and prototypes suited to the customer's application and needs.

Regards,

[nicolas]

Thank you volker, so I really was wrong with my use of contiPerf : contiPerf tests have to be dedicated to performances tests only. My examples were written only for a "mind view", and I totally agree with your post.

Nice job, thank you,

Regards,