There’s a new framework in the Java world. Quarkus. I’ve decided to compare how it behaves in relation to Apache TomEE and Spring Boot. I’m going to measure artefacts size, compilation, test and deploy times while using Java 8 and Maven.
To compare the frameworks I will create an application with a realistic set of features. It will use REST endpoints, Fault Tolerance and an in memory database with JPA. The source code is available on Github at the java-framework-compare project.
The starting point for each of the subprojects:
The Apache TomEE 8.0.0-M2 Microprofile flavour comes from the MicroProfile Starter by selecting Config and Fault Tolerance APIs.
The Spring Boot 2.1.3 application comes from the Spring Initializr page by adding web, jpa and H2 dependencies.
The Quarkus 0.11.0 app starting point is humbler. It comes from the rest-json example bundled in the quarkus-quickstarts bundle.
The baseline
All starter projects have different bundled dependencies. Just for curiosity these are the sizes, build and startup times out of the box, without an application running:
| Platform | Build Time (s) | Start Time (s) | Size (MB) |
| Apache TomEE 8.0.0-M2 | 5,454 | 3,789 | 44 |
| Spring-boot 2.1.3 | 2,248 | 2,936 | 16,7 |
| Quarkus 0.11.0 | 1,890 | 0,623 | 10,5 |
This is important to later evaluate the impact of our changes to the baseline, in a realistic project.
The application
To start, I used the implementation coming from the Quarkus rest-json quick-start example. This bundles a set of tests and rest endpoints I was able to successfully port to SpringBoot and TomEE.
The Integration tests are using RestAssured and JUnit 5 Jupiter and are equivalent across the 3 projects. They needed some tweaks on SpringBoot. Getting the random port and configuring the RestAssured serialization was not obvious.
On TomEE I had to use JUnit 4 because Arquillian, the integration test framework commonly used there, does not support JUnit 5 yet.
I’ve added Timeout and Fallback Fault tolerance features to a method. On both Quarkus and TomEE using MicroProfile and to SpringBoot using Netflix’s Hystrix. Hystrix, commonly used in SpringBoot is on maintenance mode I’m not aware of a replacement on the Spring side. As a comparison, the Microprofile Spec has added support for CompletionStage and will be doing work around reactive code soon.
I tried to use JPA with H2 database but couldn’t make it in Quarkus so I also dropped it in the others but kept all dependencies. The documentation is still sketchy and I need more research time.
Conclusions
| Platform | Build Time (s) | Build Time with tests (s) | Start Time (s) | Size (MB) |
| Apache TomEE 8.0.0-M2 | 6,178 | 15,304 | 4,993 | 44 |
| Spring-boot 2.1.3 | 3,358 | 13,348 | 6,799 | 46,9 |
| Quarkus 0.11.0 | 2,533 | 7,153 | 0,767 | 23.4 |
It looks like build time without tests increased by ~1s in all projects.
The build with tests with Quarkus takes half the time from the other 2. Artefact size is also half of the other two. Start time is almost one order of magnitude bellow. This is very significant. Please note that this start time can be further reduced by using native code with GraalVM.
SpringBoot startup time and uber war size increase very significantly (>2x) when you add real functionality to it.
TomEE starts faster than SpringBoot and the uber jar size is very stable for this use case. No Change.
The online resources on Quarkus are rare if compared with TomEE or SpringBoot leading to a lot of trial and error.
Further work
Make JPA with H2 work and measure the performance impact.
Add docker and kubernets and measure deployment times in the cloud.
Generate native artefacts for Quarkus and measure the effect.
Tech notes
Build time is calculated as the average of 3 consecutive executions of the base project using “mvn clean install -DskipTests”
Start time is calculated as the average of 3 consecutive executions of the generated uber jar/war. In the case of Quarkus it’s the runner+lib folder. Example: “java -jar target/spring-boot-0.0.1-SNAPSHOT.war”
The size is of the generated uber jar/war. In the case of Quakus it’s the runner+lib folder.
All this was executed on a Lenovo Thinkpad T460 with an Intel I7, 32GB ram and 1/3 empty 512GB SSD.
Photo taken on Saturday near Penacova, Portugal
Baptista @ Internet 
I did some performance SpringBoot, Quarkus and Microprofiles comparisons this week. I had much longer compile time in case of Quarkus. https://github.com/HotswapProjects/pingperf
Thanks for the comment.
You don’t mention the compile times in the project. Do you have some values you can post?
Mind that compile time is tight to functionality. That’s why I tried to replicate the same behaviour across all platforms.
I can extend the tables by compile time as you suggest, it could not be problem. But in our case, we are using quarkus build to native code, wich is too slow.
Thanks for the write up. I noticed a couple of things. Even with your code out of the box, I couldn’t manage to get the startup time of the Boot project above 2.5 seconds. Might be worth double checking.
Also, neither the Quarkus one nor the Boot one actually make use of JPA in the projects. Boot – as it’s sort of expecting that you *would* use it as you have added it to the classpath in the first place – still bootstraps the JPA infrastructure. If you remove the JPA related dependencies from both projects, I get to Quarkus 0.9s, Boot 2.3s. A big chunk of the Boot startup time is contributed by the bootstrap of the Cloud functionality which we’re aware of and are working on to improve on. Also, if you fancy to give the Boot 2.2 snapshots a spin, switching to that let’s the sample start in 1.4 seconds with Spring Cloud, in 0.9s without that.
Thanks for you comment Oliver.
Will work next on getting JPA working out for all projects and will update SpringBoot.
Great writeup, thanks! Hystrix can be replaced by Resilience4j (there was an announcement about it: https://spring.io/blog/2018/12/12/spring-cloud-greenwich-rc1-available-now)
Great that they will use use https://github.com/resilience4j/resilience4j. I missed that announcement.