Spring nested transactions are a bit like the black sheep of the transaction family – easily misunderstood and cast aside in documentation. Nested transactions are however incredibly useful for use cases consisting of large bulk updates and/or operating over unreliable connections. Let’s take a look at how we can bring this black sheep back …
Spring Transactions – Propagation Levels
Spring has always supported the standard 6 transaction propagation levels whose names are analogous to those in EJB land. They are specified below and can be inserted directly with the following @Transactional annotation.
@Transactional(propagation = Propagation.<SEE LIST BELOW>)
- REQUIRED (* default)
- REQUIRES_NEW
- SUPPORTS
- NOT_SUPPORTED
- MANDATORY
- NEVER
There is however one more, a 7th propagation level that not many know about and that is the NESTED transaction propagation level. Unfortunately it’s not supported by all transaction managers and it is easily confused with REQUIRES_NEW.
Why use Spring Nested Transactions?
Sometimes you have a large amount of data to push to the database (i.e 200,000 inserts). Sure you want to wrap that up in a transaction to ensure for an atomic operation however you also don’t want to have to rollback the transaction if something goes wrong on inserting record 190,000. That would be a waste of time and resources.
So what we’re going to do is break up the operation in smaller sub-units of work (20,000 inserts at a time). Then we’ll repeat that 10 times (10 x 20,000 = 200,000). The entire operation will be started in a top level transaction however the smaller sub-units of work (the 20,000) will be handled by a Spring nested transaction.
If one of those sub-units encounters a problem (runtime exception by default) then the nested transaction will roll back internally through something called savepoints but the top level transaction will be allowed to continue.
Spring Nested Transactions – Spring AOP Limitation
Not all transaction managers support nested transactions. Spring supports this out of the box only with the JDBC DataSourceTransactionManager, which is what we’ll cover. This probably has to do with the fact that nested transactions cover a minority of use cases and make a specification harder to implement across all vendors (i.e: JTA implementations).
The other limitation has to do with the way Spring AOP works in regards to proxy’ing your Spring beans. You cannot have a method with @Transactional annotation on it that calls another method in the same class that also has its own @Transactional (i.e: NESTED) and expect that to work.
There are simple workarounds for this within Spring AOP without resorting to a full blown AOP implementation such as AsjectJ AOP – which requires extra tooling and setting up. I personally am not a huge fan of the design behind those workarounds and I only present one of them (there are 3) in this post since this is exactly the limitation we come across in this use case.
Spring Boot Project Dependencies
The application I’ll use to showcase uses the following dependencies. You can also shoot over to MVP Java’s GitHub here for the source. I’ve kept the example lean and to the point so we can solely focus on Springs nested transactions.
In essence the project uses the following dependencies (Full pom.xml here).
- Spring Boot 2.1.6 with maven
- spring-boot-starter-jdbc (DataSourceTransactionManager will be used)
- spring-boot-starter-web (just need for the H2 web console)
- H2 embedded database
- com.google.guava (used to partition our list into small sub-lists which will each be past to separate NESTED transactions)
Nested Transactions in Space
For this tutorial, I’ve used a use case that I myself have come across in my professional career. Back in the days were I was working for the Canadian Space Agency, we had to build a huge list of spacecraft commands to be uploaded to the satellite everyday.
Why? Well, it needs to know what the heck it’s suppose to do and at what time. Stuff like turn this equipment on, turn this off, rotate this, take this image with these parameters for this long etc …
Me Back in 2005 at CSA in Ottawa with SCISAT-1 (in picture above). We had to generate this list of timed commands in the database. It was actually Microsoft Access if you can believe it! The list was an artifact to upload the next day to the satellite which would last be valid for the next 36 hours.
Sometimes we can’t upload the entire file to the spacecraft since the RF upload signal/connection might be too weak and so we take a best effort approach – upload smaller chunks at a time. Whatever chunk isn’t confirmed to be committed to the on-board spacecraft database, we take inventory for another attempt at a latter time and continue to the next chunk. This approach is very similar to how NESTED TRANSACTIONS function.
So in keeping with that spirit and keeping it really simple, I’ve generated a list of satellite timed commands as SQL directly in Java code and then upload them to an embedded H2 database. Sorry guys, I left the Canadian Space Agency 10 meter antenna back in Canada 😉
Spring Boot – Entry Point
The Spring Boot application really comes into play on line 23 where it prepares and attempts to upload 10 SQL commands representing the satellite timed commands via the upload method. Yes only 10 commands, why not 200,000!? Let’s keep the example simple, I don’t think you’ll like to scroll through 200,000 lines! The principle will be the same.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
package com.mvpjava.transactions; import java.util.ArrayList; import java.util.List; import java.util.Scanner; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.CommandLineRunner; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @SpringBootApplication public class Main implements CommandLineRunner { @Autowired UploaderService uploaderService; public static void main(String[] args) { SpringApplication.run(Main.class, args); } @Override public void run(String... args) throws Exception { uploaderService.upload(buildTimedCommands()); pauseApplication(); } private List<String> buildTimedCommands(){ List<String> sql = new ArrayList<>(); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('STAR_TRACKER MODE REBOOT', '2019-07-10T00:00:00Z')" ); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SAR_ANTENNA IMAGE_ON', '2019-07-10T11:22:00Z')" ); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SAR_ANTENNA STANDBY', '2019-07-10T11:29:00Z')"); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('STAR_TRACKER MODE OFF', '2019-07-10T12:00:00Z')" ); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SAR_ANTENNA IMAGE_OFF', '2019-07-10T12:22:00Z')" ); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SAR_ANTENNA STANDBY', '2019-07-10T13:29:00Z')"); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('XPNDR TX ON', '2019-07-10T14:00:00Z')" ); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('XPNDR TX OFF', '2019-07-10T14:20:00Z')" ); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SUN_SENSOR ON', '2019-07-10T14:29:00Z')"); sql.add("INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('MAG ON', '2019-07-10T14:30:00Z')"); return sql; } /* Just keep application running in order to view H2 console */ private void pauseApplication() { Scanner scanner = new Scanner(System.in); System.out.println("Press any key to stop application. Goto http://localhost:8080/h2-console URL=jdbc:h2:mem:satellitetest"); scanner.nextLine(); System.out.println("Application will close"); scanner.close(); System.exit(0); } } |
Once the upload is completed, the application is purposely paused until you hit <Enter> in the console view in order to inspect the H2 Web console at http://localhost:8080/h2-console with database URL=jdbc:h2:mem:satellitetest
H2 Embedded Database
If Spring Boot encounters a file named schema.sql in the src/main/resources directory, it will use it against the H2 Database on loading of the application. This is exactly what I’ve done to bootstrap the schema. The following DDL will allow us to specify String commands which are time stamped (AKA: Timed Commands)
1 2 3 4 5 6 7 |
DROP TABLE IF EXISTS TIMED_COMMANDS; CREATE TABLE TIMED_COMMANDS ( id INT AUTO_INCREMENT PRIMARY KEY, command VARCHAR(250) NOT NULL, time_of_execution TIMESTAMP WITH TIME ZONE DEFAULT NULL ); |
As for the actual data that goes into the TIMED_COMMANDS Table, we’ll be doing that through application code. We want to simulate inserted all those timed commands with NESTED Transactions instead of putting then in a single big transaction. This is why we’re not putting the SQL commands in src/main/resources/data.sql for Spring Boot to insert for us – defeats the purpose of the example.
Spring Boot – Nested Transactions
The idea is to start the top level transaction once the upload starts which is at the @Service level. We accomplish this by placing the @Transactional annotation on the UploaderService interface itself instead of the implementation class (which you can do either way). By default, the REQUIRED transaction propagation level is used and doesn’t have to specified as so – @Transactional(propagation = Propagation.REQUIRED)
This will start a new top level transaction which will be committed (or rolled back!) once the method completes.
1 2 3 4 5 6 7 8 9 10 11 12 |
package com.mvpjava.transactions; import java.util.List; import org.springframework.transaction.annotation.Transactional; @Transactional public interface UploaderService { public void upload(List<String> sql); } |
The implementation Class SatelliteUploaderService below will create sub-partitions (smaller sized chunks) of the List of SQL Timed Commands. We tell the Lists.partition on line 26 that we would like that each partition have a size of 3.
In our example we have 10 commands therefore it will create 3 partitions with 3 commands in each (3×3=9) as instructed but the last 10th command will be alone in its own partition list. So in the end we will end up with 4 partitions to upload in total (3+3+3+1=10). This is were your happy I didn’t use 200,000!
Here is the breakdown of the partitions with some id’s next to them for reference (i.e: Partition 2 consists of Timed Commands with id’s 4,5 and 6) …
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
//Partition 1 (to be executed in it's own NESTED Transaction) 1 - INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('STAR_TRACKER MODE REBOOT', '2019-07-10T00:00:00Z')" 2 - INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SAR_ANTENNA IMAGE_ON', '2019-07-10T11:22:00Z')" 3 - INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SAR_ANTENNA STANDBY', '2019-07-10T11:29:00Z')" //Partition 2 (to be executed in it's own NESTED Transaction) 4 - INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('STAR_TRACKER MODE OFF', '2019-07-10T12:00:00Z')" 5 - INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SAR_ANTENNA IMAGE_OFF', '2019-07-10T12:22:00Z')" 6 - INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SAR_ANTENNA STANDBY', '2019-07-10T13:29:00Z')" //Partition 3 (to be executed in it's own NESTED Transaction) 7 - INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('XPNDR TX ON', '2019-07-10T14:00:00Z')" 8 - INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('XPNDR TX OFF', '2019-07-10T14:20:00Z')" 9 - INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('SUN_SENSOR ON', '2019-07-10T14:29:00Z')" //Partition 4 (to be executed in it's own NESTED Transaction) 10 -INSERT INTO TIMED_COMMANDS (command, time_of_execution) VALUES ('MAG ON', '2019-07-10T14:30:00Z')" |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
package com.mvpjava.transactions; import java.util.List; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import com.google.common.collect.Iterables; import com.google.common.collect.Lists; @Service public class SatelliteUploaderService implements UploaderService{ private static final Logger logger = LoggerFactory.getLogger(SatelliteUploaderService.class); private final TimedCommandsDao timedCommandsDao; @Autowired public SatelliteUploaderService (TimedCommandsDao timedCommandsDao) { this.timedCommandsDao = timedCommandsDao; } @Override public void upload(List<String> timedCommands) { int DESIRED_PARTITION_SIZE = 3; List<List<String>> allPartitionedLists = Lists.partition(timedCommands, DESIRED_PARTITION_SIZE); for (List<String> subPartitionList : allPartitionedLists) { try { String[] subPartitionAsArray = Iterables.toArray(subPartitionList, String.class); timedCommandsDao.uplink(subPartitionAsArray); } catch (RuntimeException e) { logger.warn("The Transaction will commit regardless of failure in NESTED transaction", e); } } } } |
Notice on line 31, we use our Spring dependency injected dao to up-link each individual partition via the uplink method. It is the uplink method that will run in a Spring NESTED transaction (show below). You can see that the method is annotated with @Transactional(propagation = Propagation.NESTED)
Why didn’t I just place both the upload and uplink method in the same interface/Class? Wouldn’t that of been simpler and maybe even make more sense? Remember the Spring AOP limitation I described above! You can’t have the method upload call method uplink in the same Class so I opted for this design.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
package com.mvpjava.transactions; import java.util.Random; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.stereotype.Repository; import org.springframework.transaction.annotation.Propagation; import org.springframework.transaction.annotation.Transactional; @Repository public class TimedCommandsDao { private static final Logger logger = LoggerFactory.getLogger(TimedCommandsDao.class); private final JdbcTemplate jdbcTemplate; private boolean demoFailureMode; @Autowired public TimedCommandsDao(JdbcTemplate jdbcTemplate) { this.jdbcTemplate = jdbcTemplate; } @Transactional(propagation = Propagation.NESTED) public int[] uplink(String[] timedCommands) { int[] updateCounts = jdbcTemplate.batchUpdate(timedCommands); //Just for demo purposes if (demoFailureMode) { throwRandomException(); } return updateCounts; } /* * If 'nested.tx.fail' is set to true in application.properties, * it will purposely fail a random nested transaction for demo purposes */ @Value("#{new Boolean('${nested.tx.fail:false}')}") public void setDemoFailureMode(boolean demoFailureMode) { this.demoFailureMode = demoFailureMode; } /* Simulate a really bad connection */ private void throwRandomException() { if (new Random().nextInt(3) == 1) { logger.info("throwing a random exception to demo rolling back NESTED Transactions"); throw new RuntimeException(); } } } |
I’ve introduced a boolean flag to induce a random exception which will case the spring nested transaction to rollback for demo sake. This way, we can actually see the rollback in action in the logs later on.
You can see the ‘nested.txt.failed‘ boolean property is injected via the @Value annotation on line 43 above. This is actually a good time to show you the application.properties file …
1 2 3 4 5 6 7 8 9 10 |
spring.datasource.url=jdbc:h2:mem:satellitetest;DB_CLOSE_ON_EXIT=FALSE spring.h2.console.enabled=TRUE spring.h2.console.path=/h2-console spring.h2.console.settings.web-allow-others=true spring.datasource.initialization-mode=embedded logging.level.org.springframework.jdbc = TRACE #### MVP Java Properties ### #If 'nested.tx.fail' is set to true, it will purposely fail a random nested transaction for demo purposes nested.tx.fail=false |
Spring Nested Transactions – How They Work
Each time the TimedCommandsDao#uplink() method is called, a NESTED Transaction which is a sub-transaction of the top level transaction is created. A nested transaction is NOT a new independent transaction which gets committed independently of the calling transaction.
Instead, a savepoint gets created,. You can think of this as a safe place to return to if ever something bad happens, i.e: runtime exception.
If the nested transaction succeeds then the savepoint is released and execution is returned to the top level transaction where its changes will eventually be committed.
If the nested transaction fails (throws a RuntimeException) then ONLY the nested transaction will be rolled back to its savepoint and return to its original state. However the calling top level transaction is not affected (does not rollback) and keeps going on to upload the next partition of timed commands, if any. The top level transaction will still be able to commit and updates made before or after the failure which other nested transactions successfully executed.
Spring Nested Transactions – Success
Here is the console log of the application successfully uploading the entire 10 times commands. You can see the breakdown with the on-image annotations made below. Use the magnifying glass plugin provided when moving the mouse over the log image for details.
So the above console log output proves that 1 top level transaction consisting of 4 sub/Spring nested transactions are created and that only the top level transaction commits the entire upload operation.
Here is what ended up in the H2 embedded database, all 10 commands have been committed.
Spring Nested Transactions – Failure
Now we set the application property nested.tx.fail=true in the application.properties file in order to simulate random nested transaction failures. In this run we got a nested transaction failure in the first uploaded partition.
We can also confirm in the H2 database that the spacecraft timed commands are missing for id 1, 2 and 3 since they were part of the first nested transaction that rolled back.
Spring Nested Transactions – Final Advice
Although I have always found Spring nested transactions interesting, we have to keep in mind that it is the least portable solution of all the 7 transaction propagation levels.
One thing that I did not do in the example in order to keep it simple but that is vital, is to catch the exception and log as much information on the nested transaction failure as possible. Log which batch got rolled back, their id’s etc .. since you’ll need to get an inventory of this for your next attempt at a latter time.
You have to however re-throw the runtime exception after logging everything. If you forget to do this, the nested transaction won’t be rolled back to its savepoint … easy to forget, so let the runtime exception be thrown!