Understanding JMS(MDB) transaction scopes in JakartaEE application

14 November 2019

Most useful way to deal with JMS in JakartaEE application is MDB(Message Driven Beans). This type of bean acts as a JMS message listener to which messages can be delivered from either a queue or a topic.

Message Driven Bean supports only Required and NotSupported scopes as there is no incoming transaction context.

• @Required: Transaction starts before read from queue (before onMessage). All next resource calls will use this transaction context. In case transaction rollback or some RuntimeException, - message will be roll back to destination. Container acknowledges delivery after TX commit;

• @NotSupported: No transaction started. Application server acknowledges delivery on successful onMessage(). In case RuntimeException, - message will be returned to destination;

In both cases above, when message was returned to the destination, container will do redelivery according to application server configuration. For example on Wildfly it is: max-delivery-attempts and redelivery-delay.

/subsystem=messaging-activemq/server=default/address-setting=#:read-attribute(name=max-delivery-attempts)

Be careful with redelivery configuration as in case "poisen message" it can negative affect performance of your application or even server.

When you use container-managed transactions, you can invoke next MessageDrivenContext methods:

• setRollbackOnly: marks the current transaction to rollback.
• getRollbackOnly: check current transaction has been marked for rollback or not.

In case bean-managed transactions you need to manually control transaction by UserTransaction methods and you also can set the activation configuration property acknowledgeMode to Auto-acknowledge or Dups-ok-acknowledge to specify how the message received by the message-driven bean to be acknowledged.

Ok. Let's pay attention on typical use cases and potentially issues with default configurations. Often enough developers use MDB to do relatively long tasks that should be executed asynchronously. So typical message bean in this case will looks like:

@JMSDestinationDefinition(
        name = TxRequiredMessagerDrivenBean.TX_REQUIRED_QUEUE,
        interfaceName = "javax.jms.Queue"
)
@MessageDriven(activationConfig = {
    @ActivationConfigProperty(propertyName = "destinationLookup", propertyValue = TxRequiredMessagerDrivenBean.TX_REQUIRED_QUEUE),
    @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")
})
//@TransactionAttribute(TransactionAttributeType.REQUIRED)
public class TxRequiredMessagerDrivenBean implements MessageListener {

    public final static String TX_REQUIRED_QUEUE = "java:global/jms/txRequiredQueue";

    @Resource
    MessageDrivenContext messageDrivenContext;

    @Override
    public void onMessage(Message msg) {
        System.out.println("Got new message.");
        try {
            System.out.println("Hello TxRequiredMessagerDrivenBean!");
            for (int x = 1; x < 40; x++) {
                Thread.sleep(10_000l);
                System.out.println("Long transaction: " + (10 * x) + " sec.");
            }
        } catch (Exception ex) {
            System.err.println(ex);
            messageDrivenContext.setRollbackOnly();
        }
        System.out.println("Message  successfully processed");
    }
}

Now take a look to result of our bean invocation:

INFO  (Thread-1 (ActiveMQ-client-global-threads)) Got new message.
INFO  (Thread-1 (ActiveMQ-client-global-threads)) Hello TxRequiredMessagerDrivenBean!
INFO  (Thread-1 (ActiveMQ-client-global-threads)) Long transaction: 10 sec.
...
INFO  (Thread-1 (ActiveMQ-client-global-threads)) Long transaction: 290 sec.
...
WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000101:3b3ecbca:5dcc2894:13 in state  RUN
WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffff7f000101:3b3ecbca:5dcc2894:13 invoked while multiple threads active within it.
WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012381: Action id 0:ffff7f000101:3b3ecbca:5dcc2894:13 completed with multiple threads - thread Thread-1 (ActiveMQ-client-global-threads) was in progress with java.lang.Thread.sleep(Native Method) org.kostenko.example.jms.transaction.TxRequiredMessagerDrivenBean.onMessage(TxRequiredMessagerDrivenBean.java:40)
...
WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffff7f000101:3b3ecbca:5dcc2894:13 aborting with 1 threads active!
...
WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000101:3b3ecbca:5dcc2894:13
...
INFO  (Thread-3 (ActiveMQ-client-global-threads)) Got new message.
INFO  (Thread-3 (ActiveMQ-client-global-threads)) Hello TxRequiredMessagerDrivenBean!
INFO  (Thread-1 (ActiveMQ-client-global-threads)) Long transaction: 300 sec.
INFO  (Thread-3 (ActiveMQ-client-global-threads)) Long transaction: 10 sec.
INFO  (Thread-1 (ActiveMQ-client-global-threads)) Long transaction: 310 sec.
INFO  (Thread-3 (ActiveMQ-client-global-threads)) Long transaction: 20 sec.
INFO  (Thread-1 (ActiveMQ-client-global-threads)) Long transaction: 320 sec.
...
INFO  (Thread-3 (ActiveMQ-client-global-threads)) [] Long transaction: 90 sec.
INFO  (Thread-1 (ActiveMQ-client-global-threads)) [] Long transaction: 390 sec.
INFO  (Thread-1 (ActiveMQ-client-global-threads)) [] Message  successfully processed
WARN  [com.arjuna.ats.arjuna] (Thread-1 (ActiveMQ-client-global-threads)) [] ARJUNA012077: Abort called on already aborted atomic action 0:ffff7f000101:3b3ecbca:5dcc2894:13
WARN  [org.apache.activemq.artemis.ra] (Thread-1 (ActiveMQ-client-global-threads)) [] AMQ152006: Unable to call after delivery: javax.resource.spi.LocalTransactionException: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:3b3ecbca:5dcc2894:13
  Caused by: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:3b3ecbca:5dcc2894:13
...
WARN  [org.apache.activemq.artemis.core.client] (Thread-1 (ActiveMQ-client-global-threads)) [] AMQ212009: resetting session after failure

From log above we can see that:

• TransactionAttributeType.REQUIRED was used by default;
• Started transaction was canceled by default timeout in 300 sec;
• Container detected TX cancellation - did message redelivery and MDB starts new transaction;
• Old thread continue work with first message processing, but the transaction already is not active;

So, keep this behavior in mind if you planning to play with long tasks and Message Driven Beans then depends on requirements few solutions is possible here:

• Avoid long transaction usage. If long TX is not necessary - always use TransactionAttributeType.NOT_SUPPORTED
• Increase default transaction timeout
• Use an annotation @ActivationConfigProperty(propertyName="transactionTimeout", propertyValue="xxx") to specify custom transaction timeout for MDB like:

@MessageDriven(activationConfig = {
    @ActivationConfigProperty(propertyName = "destinationLookup", propertyValue = TxRequiredMessagerDrivenBean.TX_REQUIRED_QUEUE),
    @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")
    @ActivationConfigProperty(propertyName = "transactionTimeout", propertyValue="500")
})

Source code of test application available on GitHub

Comments

---

Never use Java primitives for math calculation

08 November 2019

This is well known stuff about representation of Floating point types in java, but time from time each developer can forget about

@Test
public void primitiveTest() {

    // right way:
    BigDecimal bgDcml = BigDecimal.valueOf(100000f).multiply(BigDecimal.valueOf(750f)).multiply(BigDecimal.valueOf(15f)).setScale(0);
    BigDecimal bgDcml2 = BigDecimal.valueOf(0.04d).multiply(BigDecimal.valueOf(15.0d));

    // wrong way:
    float flt = 100000f * 750f * 15f;
    float flt2 = 0.04f * 15.0f;

    System.out.println("BigDecimal (correct): " + bgDcml);
    System.out.println(String.format("Float (incorrect): %s (%s)", flt, new BigDecimal(flt)));
    System.out.println("BigDecimal (correct): " + bgDcml2);
    System.out.println(String.format("Float (incorrect): %s", flt2));
}

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
BigDecimal (correct): 1125000000
Float (incorrect): 1.12499994E9 (1124999936)
BigDecimal (correct): 0.600
Float (incorrect): 0.59999996

Comments

---

Jakarta Batch garbage collection with Jberet

04 November 2019

To implement Jakarta Batch Specification Wildfly uses JBeret as implementation of JSR 352 (Batch Applications for the Java Platform). During processing, last one persists some working data to the memory, JDBC or another repository depends on configuration.

In case intensive usage it can collect lot of data and as result provoke some application server performance issues. To cleanup unwanted job data you can design your own solution or use provided by Jberet org.jberet.repository.PurgeBatchlet in standard way:

<job id="batchGarbageCollector" xmlns="http://xmlns.jcp.org/xml/ns/javaee"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/jobXML_1_0.xsd"
     version="1.0">
    <step id="batchGarbageCollector.step1">
        <batchlet ref="org.jberet.repository.PurgeBatchlet">
            <properties>
                <property name="sqlFile" value="#{jobParameters['sqlFile']}"/>
            </properties>
        </batchlet>
    </step>
</job>

From the documentation PurgeBatchlet supports rich set of properties and looks pretty good, BUT on practice lot of them does not work as expected. I tried numberOfRecentJobExecutionsToKeep and batchStatuses on both WF repositories (inMemory and JDBC) and on both of them it does not work for me.

I am getting output like below, but garbage still was with me :(

...
INFO  [org.jberet] (Batch Thread - 8) [] JBERET000023: Removing javax.batch.runtime.JobExecution 35256804
INFO  [org.jberet] (Batch Thread - 8) [] JBERET000023: Removing javax.batch.runtime.JobExecution 35256806
...

Good news is, that in case using JDBC repository job parameter sqlFile works as expected and PurgeBatchlet executes provided SQL... without any log outputs.

Source code of test application available on GitHub

Comments

---

Migration from Wildfly 10 to Wildfly 18

22 October 2019

Usually vendors declares easy migration, provides how-to documentation and even migration tools. But depends on complexity of your application you can stuck with some compatibility issues. Below i will explain my Wildfliy 10 to Wildfliy 18 migration steps.

First of all you can use provided by WF migration tool to migrate configuration files and compatible modules to the needed release:

git clone https://github.com/wildfly/wildfly-server-migration.git
cd ./wildfly-server-migration/
mvn clean install
cd ./dist/standalone/target/
unzip jboss-server-migration-1.8.0.Final-SNAPSHOT.zip
cd ./jboss-server-migration

./jboss-server-migration.sh -s /opt/wildfly-10.1.0.Final -t /opt/wildfly-18.0.0.Final/

Now let's see application level migration issues:

Issue #1:

Caused by: java.lang.IllegalArgumentException: org.hibernate.hql.internal.ast.QuerySyntaxException: Invalid path: 'org.kostenko.STATUS.ACTIVE'

Error above related to the Hibernate 5.2 performance improvement that avoids unnecessary calls to Class.forName(). Solution here is using Java Naming conventions for a constant. (for example rename STATUS to Status) or in case using non-conventional Java constants set the

<property name="hibernate.query.conventional_java_constants" value="false"/>

in your persistence.xml

Issue #2:

java.lang.IllegalArgumentException: ArquillianServletRunner not found. Could not determine ContextRoot from ProtocolMetadata, please contact DeployableContainer developer.

In case using Arquillian you need just update version org.wildfly.arquillian:wildfly-arquillian-container-managed to version 2.2.0.Final

Issue #3:

org.jboss.weld.exceptions.UnsupportedOperationException:
  at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:49)

Seems to old BUG happened and Weld does not support Java 8 default methods completely. So, pity but same refactoring here is needed.

Issue #4:

WFLYRS0018: Explicit usage of Jackson annotation in a JAX-RS deployment; the system will disable JSON-B processing for the current deployment. Consider setting the 'resteasy.preferJacksonOverJsonB' property to 'false' to restore JSON-B.
...
javax.ws.rs.client.ResponseProcessingException: javax.ws.rs.ProcessingException: RESTEASY008200: JSON Binding deserialization error  
  at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.extractResult(ClientInvocation.java:156)  
  at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:473)  
  at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.get(ClientInvocationBuilder.java:195)  

Since latest versions, WF uses org.eclipse.yasson as JSON-B provider. It can provoke some compatibility problems in case using different implementations. Solution here is refactoring according to the JSON-B specification or excluding resteasy-json-binding-provider from application class loader by providing WEB-INF/jboss-deployment-structure.xml:

<?xml version="1.0" encoding="UTF-8"?>
<jboss-deployment-structure>
    <deployment>
        <exclusions>
            <module name="org.jboss.resteasy.resteasy-json-binding-provider"/>
        </exclusions>
    </deployment>
</jboss-deployment-structure>

or in case using EARs, do exclusion from submodules like

<?xml version="1.0" encoding="UTF-8"?>
<jboss-deployment-structure>
    <deployment>
        <exclusions>
            <module name="org.jboss.resteasy.resteasy-json-binding-provider"/>
        </exclusions>
    </deployment>
    <sub-deployment name="module-1.jar">
        <exclusions>
            <module name="org.jboss.resteasy.resteasy-json-binding-provider"/>
        </exclusions>
    </sub-deployment>    
</jboss-deployment-structure>

Issue #5:
According to the fixed Hibernate BUG, for now JPA call setMaxResult(0) returns empty List instead of ALL elements in previous versions. So, just check it and do some refactoring if needed.

Comments

---

JAX-RS Client ThreadPool leak

04 October 2019

Recently got resource(ThreadPool\Thread) leak with JAX-RS Client implementation on WF10.0.1 (RestEasy).

From the dump above we can see, that pool number is extremely height, the same time thread number is always 1. That means that some code uses Executors.new*, which returns java.util.concurrent.ThreadPoolExecutor using the DefaultThreadFactory.

Actually in this situation, it is ALL than we can see from thread and heap dumps when debugging leak like above. Because in case classes containing these executors was garbage collected, the executors get orphaned (but are still alive and uncollectable), making it difficult/impossible to detect from a heap dump where the executors came from.

Lesson #1 is: Doing Executors.new*, would be nice to little bit think about guys who will support your code and provide non default thread names with custom ThreadFactory like :)

ExecutorService es = Executors.newCachedThreadPool(new CustomThreadFactory());

...

class CustomThreadFactory implements ThreadFactory {

    @Override
    public Thread newThread(final Runnable r) {
        return new Thread(r, "nice_place_for_helpful_name");
    }
}

So, after many times of investigation and "heap walking"(paths to GC root) i found few Executors$DefaultThreadFactory like

what made me see the code with REST services invocations. Something like

public void doCall() {
    Client client = ClientBuilder.newClient();
    Future<Response> future = client.target("http://...")
                                 .request()
                                 .async().get();
}

According to WF10 JAX-RS implementation each newClient() will build ResteasyClient that uses ExecutorService asyncInvocationExecutor to do requests and potentially it is can be the reason of the leak.

Lesson #2 is: Always! do close() client after usage. Check that implementation closes connection and shutdowns ThreadPool in case errors (timeouts, socket resets, etc).

Lesson #3 is: Try to construct only a small number of Client instances in the application. Last one is still bit unclear from the pure JakartaEE application point of view, as it not works as well in multi-threaded environment. (Invalid use of BasicClientConnManager: connection still allocated. Make sure to release the connection before allocating another one.)

P.S. Many thanks for JProfiler tool trial version to make me happy with ThreadDump walking.

Comments

---

• Previous
Page: 6/12 (archive)
• Next

© 2019 | Mixed with Bootstrap v3.1.1 | Baked with JBake v2.6.4