Spring Boot is a very popular framework for Java enterprise applications. One common method of integration with internal or external applications is through HTTP REST connections. We’ve been upgrading from RestTemplate to Java NIO-based WebClient, which can significantly improve application performance by allowing concurrency when calling REST service endpoints. The advantages of WebClients are as follows:
- Competition: WebClient allows handling multiple connections simultaneously without blocking threads, leading to better concurrency.
- Asynchronous: Asynchronous programming allows an application to perform other tasks while waiting for I/O operations to complete, improving overall efficiency.
- Performance: Non-blocking I/O can handle more connections with fewer threads, reducing the resources needed to handle concurrent requests.
Although the performance improved, however with the same number of concurrent connections, the WebClient crashed OutOfMemoryError. We will analyze WebClient crashing issues along with ways to troubleshoot and fix them.
Spring RestTemplate to WebClient Upgrade
To take advantage of NIO’s advantages, such as concurrency and asynchronous processing, we upgraded the rest client call from Spring RestTemplate on the WebClient, as shown below.
Spring RestTemplate
public void restClientCall(Integer id, String url,String imagePath)
// Create RestTemplate instance
RestTemplate restTemplate = new RestTemplate();
// Prepare the image file
File imageFile = new File(imagePath);
// Prepare headers
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
// Prepare the request body
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new org.springframework.core.io.FileSystemResource(imageFile));
// Create the HTTP entity with headers and the multipart body
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);
System.out.println("Starting to post an image for Id"+id);
// Perform the POST request
ResponseEntity<String> responseEntity = restTemplate.postForEntity(url, requestEntity,
String.class);
// Print the response status code and body
System.out.println("Response Id "+id +":"+ responseEntity.getBody());
System.out.println(" Time: " + LocalTime.now());
On to the next one Spring WebClient as below:
public void webHeavyClientCall(Integer id,String url, String imagePath)
// Create a WebClient instance
WebClient webClient = WebClient.create();
// Prepare the image file
File imageFile = new File(imagePath);
// Perform the POST request with the image as a part of the request body
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new FileSystemResource(imageFile));
System.out.println("Image upload started "+id);
webClient.post().uri(url).contentType(MediaType.MULTIPART_FORM_DATA).body(BodyInserters.fromMultipartData
(body)).retrieve().bodyToMono(String.class).subscribe(response ->
System.out.println("Response Id"+id+ ":" + response);
);
WebClient Result OutOfMemoryError
When we ran both programs in OpenJDK 11. The program that used the NIO-based Spring WebClient resulted in ‘java.lang.OutOfMemoryError: Direct buffer memory’ after several repetitions, while it is spring RestTemplate based program was successfully completed. Below is the output of the NIO based Spring WebClient. You can notice ‘java.lang.OutOfMemoryError’ is registered.
Starting to post an image for Id0
Starting to post an image for Id1
Starting to post an image for Id2
Starting to post an image for Id3
Starting to post an image for Id4
Starting to post an image for Id5
Starting to post an image for Id6
Starting to post an image for Id7
Starting to post an image for Id8
Starting to post an image for Id9
Starting to post an image for Id10
Starting to post an image for Id11
Starting to post an image for Id12
Starting to post an image for Id13
Starting to post an image for Id14
2023-12-06 17:21:46.730 WARN 13804 --- [tor-http-nio-12] io.netty.util.concurrent.DefaultPromise : An
exception was thrown by reactor.ipc.netty.FutureMono$FutureSubscription.operationComplete()
reactor.core.Exceptions$ErrorCallbackNotImplemented:
io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175) ~[na:na]
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[na:na]
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318) ~[na:na]
at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242) ~[na:na]
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:164) ~[na:na]
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[na:na]
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496) ~[na:na]
at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:418) ~[netty-
transport-4.1.23.Final.jar!/:4.1.23.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) ~[netty-
transport-4.1.23.Final.jar!/:4.1.23.Final]
... 18 common frames omittedTroubleshooting ‘OutOfMemoryError: Direct buffer memory’
To solve this problem, we used the yCrash tracker. This tool can predict outages before they occur in a production environment. After predicting an outage in the environment, it captures 360° troubleshooting artifacts from your environment, analyzes them, and instantly generates a root cause analysis report. The artifacts it records include garbage collection log, thread dump, heap swap, netstat, vmstat, iostat, top, top -H, dmesg, kernel parameters and disk usage….
You can register here and start using the free level of this tool.
The yCrash server analyzed the Spring Boot Rest Client and gave clear indications of referral problems. Below is a summary of the incident report generated by yCrash for the SpringBoot WebClient application. You may notice that yCrash clearly indicates the error with the necessary troubleshooting recommendations.
Garbage collection analysis report
A Garbage Collection (GC) analysis report by yCrash revealed that full GCs were run consecutively (see screenshot below). When the GC starts, the entire application stops and no transactions will be processed. The whole application would stop responding. We noticed an unresponsiveness before the SpringBoot WebClient application crashed OutOfMemoryError.
Log analysis reporting OutOfMemoryError: Direct buffer memory
The yCrash application log analysis report revealed that the application suffers from ‘java.lang.OutOfMemoryError: Direct buffer memory’ (see screenshot below) which causes the app to crash.
Why does Spring WebClient suffer from OutOfMemoryError?
Figure 4: RestTemplate objects stored in the Others region of source memory
Spring WebClient is developed based on Java NIO technology. In Java NIO, objects are stored in the ‘Direct Buffer Memory’ region of the JVM native memory, while RestTemplate objects are stored in the ‘others’ region of the JVM native memory. There are different memory regions in the JVM. To know more about them, you can watch this video clip.
When we executed the above two programs, we set the direct buffer memory size to 200k (ie –XX:MaxDirectMemorySize=200k). This size was sufficient for Spring RestTemplate, because objects were never stored in this region, on the other hand it was not sufficient for Spring WebClient. So Spring WebClient suffered from java.lang.OutOfMemoryError: Direct buffer memory.
Increasing -XX:MaxDirectMemorySize
After identifying this problem we increased the direct memory size to a higher value using the JVM argument -XX:MaxDirectMemorySize=1000k. After this change, the Spring WebClient program worked perfectly fine without any problems.
Starting to post an image for Id0
Starting to post an image for Id1
Starting to post an image for Id2
Starting to post an image for Id3
Starting to post an image for Id4
Starting to post an image for Id5
Starting to post an image for Id6
Starting to post an image for Id7
Starting to post an image for Id8
Starting to post an image for Id9
Starting to post an image for Id10
Starting to post an image for Id11
Starting to post an image for Id12
Starting to post an image for Id13
Starting to post an image for Id14
Starting to post an image for Id15
Starting to post an image for Id16
Starting to post an image for Id17
Starting to post an image for Id18
Starting to post an image for Id19
Response Id11:Image uploaded successfully!
Response Id4:Image uploaded successfully!
Response Id1:Image uploaded successfully!
Response Id18:Image uploaded successfully!
Response Id2:Image uploaded successfully!
Response Id3:Image uploaded successfully!
Response Id6:Image uploaded successfully!
Response Id5:Image uploaded successfully!
Response Id10:Image uploaded successfully!
Response Id13:Image uploaded successfully!
Response Id15:Image uploaded successfully!
Response Id8:Image uploaded successfully!
Response Id17:Image uploaded successfully!
Response Id9:Image uploaded successfully!
Response Id7:Image uploaded successfully!
Response Id0:Image uploaded successfully!
Response Id16:Image uploaded successfully!
Response Id14:Image uploaded successfully!
Response Id19:Image uploaded successfully!
Response Id12:Image uploaded successfully!Conclusion
In this post, we discussed the OutOfMemoryError issue we faced while upgrading from Spring RestTemplate on the Java NIO-based WebClient. We also shared the diagnostic approach we took and then the solution to the problem. We hope it will be of use to you.