JAX-RS Client ThreadPool leak

04 October 2019

Recently got resource(ThreadPool\Thread) leak with JAX-RS Client implementation on WF10.0.1 (RestEasy).
jax-rs thread leak

From the dump above we can see, that pool number is extremely height, the same time thread number is always 1. That means that some code uses Executors.new*, which returns java.util.concurrent.ThreadPoolExecutor using the DefaultThreadFactory.

Actually in this situation, it is ALL than we can see from thread and heap dumps when debugging leak like above. Because in case classes containing these executors was garbage collected, the executors get orphaned (but are still alive and uncollectable), making it difficult/impossible to detect from a heap dump where the executors came from.

Lesson #1 is: Doing Executors.new*, would be nice to little bit think about guys who will support your code and provide non default thread names with custom ThreadFactory like :)

ExecutorService es = Executors.newCachedThreadPool(new CustomThreadFactory());

...

class CustomThreadFactory implements ThreadFactory {

    @Override
    public Thread newThread(final Runnable r) {
        return new Thread(r, "nice_place_for_helpful_name");
    }
}

So, after many times of investigation and "heap walking"(paths to GC root) i found few Executors$DefaultThreadFactory like

jax-rs thread leak

what made me see the code with REST services invocations. Something like

public void doCall() {
    Client client = ClientBuilder.newClient();
    Future<Response> future = client.target("http://...")
                                 .request()
                                 .async().get();
}

According to WF10 JAX-RS implementation each newClient() will build ResteasyClient that uses ExecutorService asyncInvocationExecutor to do requests and potentially it is can be the reason of the leak.

Lesson #2 is: Always! do close() client after usage. Check that implementation closes connection and shutdowns ThreadPool in case errors (timeouts, socket resets, etc).

Lesson #3 is: Try to construct only a small number of Client instances in the application. Last one is still bit unclear from the pure JakartaEE application point of view, as it not works as well in multi-threaded environment. (Invalid use of BasicClientConnManager: connection still allocated. Make sure to release the connection before allocating another one.)

P.S. Many thanks for JProfiler tool trial version to make me happy with ThreadDump walking.


Migration from JEE to JakartaEE

30 September 2019

As you probably know Java EE was moved from Oracle to the Eclipse Foundation where will evolve under the Jakarta EE brand. Sept. 10, 2019 Jakarta EE Full Platform and Web Profile specifications was released by Eclipse Foundation during JakartaOne Livestream. Few days later Wildfly declared that WildFly 17.0.1 has passed the Jakarta EE 8 TCK and certification request has been approved by the Jakarta EE Spec Committee. So, now WildFly is a Jakarta EE Full platform compatible implementation.

Let's do migration of typical gradle EE project to the Jakarta EE and look how hard is it. Current JakartaEE version 8.0.0 is fully compatible with JavaEE version 8.0, that means no need to change project sources, just update dependency from javax:javaee-api:8.0 to jakarta.platform:jakarta.jakartaee-api:8.0.0

updated build.gradle:

apply plugin: 'war'
dependencies {
    providedCompile "jakarta.platform:jakarta.jakartaee-api:8.0.0"
}

That is it! Application builds and works well under WF17.0.1

Source code of demo application available on GitHub


Wildfly JMX connection problems (so slow and terminates)

19 August 2019

JMX (Java Management Extensions ) - is a technology that provide us possibility to monitoring applications (application servers) by MBeans (Managed Bean) objects.
List of supported MBeans can be obtained by JConsole tool that already included to JDK. As JMX does not provide strong defined communication protocol, - implementations can be different depends on vendor.
For example, to connect to Wildfly Application Server you need to use included in distribution jconsole.sh script:

<WFLY_HOME>/bin/jconsole.sh

or add <WFLY_HOME>/bin/client/jboss-client.jar to classpath:

jconsole J-Djava.class.path=$JAVA_HOME\lib\tools.jar;$JAVA_HOME\lib\jconsole.jar;jboss-client.jar

By default, Wildfly uses timeout = 60s for remote JMX connections, after that connection will terminated:
jconsole terminated connection
To change default timeout value, use org.jboss.remoting-jmx.timeout property:

./jconsole.sh -J-Dorg.jboss.remoting-jmx.timeout=300

But increasing timeouts, is not always good solution. So, lets search for the reason of slowness. To construct list of MBeans, jconsole recursively requests ALL MBeans, that can be extremely slow in case many deployments and many loggers. (Reported issue: WFCORE-3186). Partial solution here is reducing count of log files by changing rotating type from periodic-size-rotating-file-handler to size-rotating-file-handler.

Other reason of extremely slowness can be Batch subsystem (JBeret). Last one stores a lot of working information in their tables (in memory or on remote DB, depends on configuration). If this tables big enough - it can negative affect performance of server. So, if you, no need for this data then just cleanup this stuff periodically. (for example, every redeploy in case you do it often enough):

TRUNCATE TABLE PARTITION_EXECUTION CASCADE;
TRUNCATE TABLE STEP_EXECUTION CASCADE;
TRUNCATE TABLE JOB_EXECUTION CASCADE;
TRUNCATE TABLE JOB_INSTANCE CASCADE;  

From other point of view, obtaining ALL MBeans is not good decision as well. So, just use tooling that allows to find MBeans by path.


Jakarta EE application multi module gradle template

08 August 2019

In this post i will share simple and useful gradle template to organize multi module Jakarta EE application. We will implement typical one which consists from REST controller (module1) and some main logic (module2). Big picture of our application architecture is:

EE multi module application

So, lets do initialization of project with next gradle template:
settings.gradle:

rootProject.name = 'ee-application-multi-module-gradle-template'
include 'module1'
include 'module2:module2-api', 'module2:module2-core'

root build.gradle:

defaultTasks 'clean', 'build'
subprojects {
    ext.libraryVersions = [
        javaee                  : '8.0',
    ]
    defaultTasks 'clean', 'build'
    repositories {
        jcenter()
    }
}

Above, we described initial application structure, where module1 is flat sub project for our controller and module2 is our main logic which consists from API and Core sub projects. As controller will use main logic API and we decided to separate application to modules (that means no big enterprise archive) - our sub projects should be simple enough:

module1 build.gradle:

apply plugin: 'war'
dependencies {
    compile project(':module2:module2-api')
    providedCompile "javax:javaee-api:${libraryVersions.javaee}"
}

module2:module2-api:

apply plugin: 'java'
dependencies {
}

module2:module2-core:

apply plugin: 'war'
dependencies {
    compile project(':module2:module2-api')
    providedCompile "javax:javaee-api:${libraryVersions.javaee}"
}

Actually, that's it!
Now we can implement our controller like:

@Path("/")
@Stateless
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public class TestEndpoint {

    @EJB(lookup = TestService.TEST_SERVICE_JNDI)
    TestService testService;

    @GET
    @Path("/test")
    public Response test() {
        SomethingDto something = testService.doSomething();
        return Response.ok().entity(something.getMessage()).build();
    }

In turn, main logic API contents from Interface and DTO:
TestService.java:

public interface TestService {

  String TEST_SERVICE_NAME = "test-service";
  String TEST_SERVICE_JNDI ="java:global/module2-core/" + TEST_SERVICE_NAME;

  SomethingDto doSomething();
}

SomethingDto.java:

public class SomethingDto implements Serializable{
  ...
}

In the end, main logic Core contents from the logic that implements API:
TestServiceImpl.java

@Remote(TestService.class)
@Stateless(name = TestService.TEST_SERVICE_NAME)
public class TestServiceImpl implements TestService {

    @PersistenceContext
    EntityManager entityManager;

    @Override
    public SomethingDto doSomething() {
        TestEntity entity = entityManager.find(TestEntity.class, Long.MAX_VALUE);
        return new SomethingDto("Hello Jakarta EE world!");
    }
}

Described Jakarta EE application architecture allows us enjoy all power of EE with absolutely transparent inter module interactions and, the same time, stay close to micro service design - as we have no limits with using one container for all modules.

Source code of this demo available on GitHub