Avoid pause/restart for application context in the TestContext framework

[bdeneuter]

We noticed that our tests which use @SpringBootTest run much slower after upgrading to Spring Boot 4.0.

We noticed that the application context is reused between tests from a single test class.

But the application context is paused and restarted between test classes while all test classes are using the same context. This is enforced by extending an abstract class which contains the configuration.

We enabled the DEBUG level to make sure only 1 context is in the cache during the build:

Spring test ApplicationContext cache statistics: [DefaultContextCache@1edd726c size = 1, maxSize = 32, contextUsageCount = 1, parentContextCount = 0, hitCount = 48, missCount = 1, failureCount = 0]

We noticed that SpringExtension.afterAll will call testContextManager.afterTestClass(). This will mark the context as unused and as a consequence will pause the context. When the new test runs it will fetch the same context from the cache and restart it.

This is the stack trace causing the context to be paused:

at org.springframework.test.context.cache.DefaultContextCache.unregisterContextUsage(DefaultContextCache.java:226)
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.unregisterContextUsage(DefaultCacheAwareContextLoaderDelegate.java:225)
at org.springframework.test.context.support.DefaultTestContext.markApplicationContextUnused(DefaultTestContext.java:155)
at org.springframework.test.context.TestContextManager.afterTestClass(TestContextManager.java:559)
at org.springframework.test.context.junit.jupiter.SpringExtension.afterAll(SpringExtension.java:183)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor$$Lambda/0x00000fc002f0bc00.invoke(Unknown Source:-1)
at org.junit.jupiter.engine.descriptor.CallbackSupport.lambda$invokeAfterCallbacks$1(CallbackSupport.java:49)
at org.junit.jupiter.engine.descriptor.CallbackSupport$$Lambda/0x00000fc002eeec00.execute(Unknown Source:-1)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:74)
at org.junit.jupiter.engine.descriptor.CallbackSupport.lambda$invokeAfterCallbacks$0(CallbackSupport.java:49)
at org.junit.jupiter.engine.descriptor.CallbackSupport$$Lambda/0x00000fc002eee800.accept(Unknown Source:-1)
at org.junit.platform.commons.util.CollectionUtils.forEachInReverseOrder(CollectionUtils.java:213)
at org.junit.jupiter.engine.descriptor.CallbackSupport.invokeAfterCallbacks(CallbackSupport.java:48)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeAfterAllCallbacks(ClassBasedTestDescriptor.java:493)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.after(ClassBasedTestDescriptor.java:271)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.after(ClassBasedTestDescriptor.java:88)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:186)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda/0x00000fc002effc00.execute(Unknown Source:-1)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:74)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$1(NodeTestTask.java:186)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda/0x00000fc001324c00.invoke(Unknown Source:-1)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$0(NodeTestTask.java:164)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda/0x00000fc001324800.execute(Unknown Source:-1)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:74)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:163)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:116)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda/0x00000fc001326400.accept(Unknown Source:-1)
at java.util.ArrayList.forEach(ArrayList.java:1604)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:42)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$2(NodeTestTask.java:180)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda/0x00000fc001325000.execute(Unknown Source:-1)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:74)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$1(NodeTestTask.java:166)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda/0x00000fc001324c00.invoke(Unknown Source:-1)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$0(NodeTestTask.java:164)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda/0x00000fc001324800.execute(Unknown Source:-1)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:74)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:163)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:116)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:36)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:52)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:58)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.executeEngine(EngineExecutionOrchestrator.java:246)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.failOrExecuteEngine(EngineExecutionOrchestrator.java:218)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:179)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:66)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda/0x00000fc001305c00.accept(Unknown Source:-1)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:157)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:65)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:125)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:93)
at org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:48)
at org.junit.platform.launcher.core.InterceptingLauncher.lambda$execute$0(InterceptingLauncher.java:41)
at org.junit.platform.launcher.core.InterceptingLauncher$$Lambda/0x00000fc001229000.proceed(Unknown Source:-1)
at org.junit.platform.launcher.core.ClasspathAlignmentCheckingLauncherInterceptor.intercept(ClasspathAlignmentCheckingLauncherInterceptor.java:25)
at org.junit.platform.launcher.core.InterceptingLauncher.execute(InterceptingLauncher.java:40)
at org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:48)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:135)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:110)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:104)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:64)
at java.lang.invoke.LambdaForm$DMH/0x00000fc0011ec000.invokeInterface(LambdaForm$DMH:-1)
at java.lang.invoke.LambdaForm$MH/0x00000fc0011ec800.invoke(LambdaForm$MH:-1)
at java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder:-1)
at jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154)
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.lang.reflect.Method.invoke(Method.java:565)
at org.gradle.internal.dispatch.MethodInvocation.invokeOn(MethodInvocation.java:77)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:28)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:19)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:88)
at jdk.proxy2.$Proxy6.stop(Unknown Source:-1)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:194)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:126)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:122)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:72)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

[bart-deneuter-hs]

I'm adding a sample project that reproduces the bug. I created an abstract test class with the @SpringBootTest annotation and 2 test classes. I created also a Spring Bean implementing SmartLifecycle. In both tests I check that the context was not restarted by asking the SmartLifecycle Spring bean if it was stopped. I would expect that the context is not stopped between test as both tests use the same context

context-restart.zip

[jhoeller]

If an application context is not in use by a test, it gets stopped indeed, even if the subsequent test obtains the same context again. This rarely happens in concurrent runs but is rather common in sequential test runs. From a 7.0 perspective, this is by design; we did not expect that to cause significant overhead since stop should be a quick step (most importantly, stopping the context's managed thread pools).

Which Lifecycle beans actually cause the overhead in your case? They could implement SmartLifecycle#isPauseable() to return false if they preferred to be ignored in such a restart scenario. Otherwise, we expect stop() and start() to be quick enough operations that they can even be called in immediate succession without significant overhead. That's the initial design perspective that we took there, at least.

So I see various ways to improve the out-of-the-box experience here:

• We could declare pauseable=false in various components that are expensive to stop/start. This is definitely advisable for custom components but can be considered for certain Spring-included components as well.
• We could try to find a solution for immediate reuse of the same context where we only actually stop the context if the next context taken from the cache is a different one. However, this is not straightforward to design.
• We could support an annotation-based declaration at the test class level that prevents its application context from getting paused.

[bart-deneuter-hs]

Starting and stopping our Kafka consumers seems to cause most overhead.
We are not running the tests in parallel as we have some Spring beans which are mocks from Mockito and they are reset between each test. Running in parallel would cause flaky tests as they would interact with each other.
An annotation at the test class level would work as we use a common abstract test class on which we can declare this annotation