Performance test for Hibernate Validator

The tests are written as JMH benchmarks. They can be developed and executed as usual in your IDE. However, they are not executed during regular build as unit tests when a mvn test is running. Instead, to execute the performance tests, one should build a jar file with one of the profiles mentioned below and run it.

Profiles

To allow performance testing of different Hibernate Validator versions there are multiple profiles configured. Choosing a profile executes the tests against the specified Hibernate Validator or BVal version, respectively. The defined profiles are:

• hv-current (Hibernate Validator 6.1.0-SNAPSHOT)
• hv-6.0 (Hibernate Validator 6.0.15.Final)
• hv-5.4 (Hibernate Validator 5.4.3.Final)
• hv-5.3 (Hibernate Validator 5.3.4.Final)
• hv-5.2 (Hibernate Validator 5.2.4.Final)
• hv-5.1 (Hibernate Validator 5.1.3.Final)
• hv-5.0 (Hibernate Validator 5.0.1.Final)
• hv-4.3 (Hibernate Validator 4.3.0.Final)
• hv-4.2 (Hibernate Validator 4.2.0.Final)
• hv-4.1 (Hibernate Validator 4.1.0.Final)
• bval-1.1 (Apache BVal 1.1.2)

Executing the performance tests

Some tips before you start:

• These tests are not there to measure absolute execution times! Instead their value lies in the ability to compare against different versions of Hibernate Validator using the same tests. This allows to detect and address performance regressions.
• You need to specify a profile!

Using the maven build

The following command line will run all performance tests listed in the main method of TestRunner class:

mvn clean package -Dvalidator=hv-current
java -jar target/hibernate-validator-performance-hv-current.jar

It will generate a set of reports from each test execution. Currently, all test results information are inside the following generated file:

target/jmh-results.json

Profiling

List of available profilers:

If you want to run one of those profilers - pass it as parameter when running a jar file. For example:

java -jar target/hibernate-validator-performance-hv-current.jar -prof org.openjdk.jmh.profile.StackProfiler

To run a specific benchmark:

java -jar target/hibernate-validator-performance.jar CascadedValidation

Creating reports for all major Hibernate Validator versions

mkdir reports
for impl in "bval-1.1.2" "hv-5.4" "hv-current"; do
    mvn -Dvalidator=${impl} package ; java -jar target/hibernate-validator-performance-${impl}.jar -rff reports/${impl}-jmh-results.json
done

Existing tests

At the moment the following benchmarks are defined:

SimpleValidation

A simple bean with a random number of failing constraints gets initialized and validated. The test is once executed with a shared ValidatorFactory and once the factory is recreated on each invocation.

CascadedValidation

Simple bean with cascaded validation which gets executed over and over.

CascadedWithLotsOfItemsValidation

Validation of a bean containing a lot of beans to cascade to.

CascadedWithLotsOfItemsAndMoreConstraintsValidation

This test has a few more constraints than the previous one, allowing to test our hypothesis in more realistic situation.

StatisticalValidation

A number of _TestEntity_s is created where each entity contains a property for each built-in constraint type and also a reference to another TestEntity. All constraints are evaluated by a single ConstraintValidator implementation which fails a specified percentage of the validations.

Adding more tests

When adding new performance tests, note the sources structure of this module:

• java/: main sources that contain the test runner
• jakarta/: performance tests based on the jakarta.validation namespace and that are applicable to the Jakarta Validation spec implementation.
• javax/: performance tests based on the javax.validation namespace and that are applicable to the Java Bean Validation spec implementation for versions 1.0 and 1.1.
• javax-bv2/ performance tests based on the javax.validation namespace and that are applicable to the Java Bean Validation spec implementation for version 2.0.

Hence, when a new test is added, consider adding it to the jakarta and javax directories depending on the tested functionality.