You Need To Use Queueable Finalizers
Table of Contents:
In the Summer 21 release, finalizers for System.Queueable classes were made generally available. Since then, they’ve received very little fanfare, despite being one of the most groundbreaking features introduced to the standard platform since Queueables themselves. Let’s do a deep dive on why the System.Finalizer interface is amazing, and why if you haven’t been using Finalizers on every Queueable, you’re missing out!
Getting Started
A Simple Test
Easing in to talking about Finalizers is simple primarily because implementing the interface is trivial:
public class FinalizerExample implements System.Finalizer {
public void execute(System.FinalizerContext fc) {
// do stuff here
}
}
// and then in a Queueable class:
public class QueueableExample implements System.Queueable {
public void execute(System.QueueableContext qc) {
System.attachFinalizer(new FinalizerExample());
// do async stuff here
}
}Those are the basics. But what does that get you? Let’s write a test to show you something that wouldn’t have been possible previously:
public class QueueableExample implements System.Queueable {
public Boolean shouldEnqueueAgain = false;
public void execute(System.QueueableContext qc) {
System.attachFinalizer(new FinalizerExample());
// do async stuff here
if (this.shouldEnqueueAgain) {
System.enqueueJob(this);
}
}
}
@IsTest
private class QueueableExampleTest {
@IsTest
static void enqueuesMultipleTimesInTheSameTransaction() {
QueueableExample example = new QueueableExample();
example.shouldEnqueueAgain = true;
Test.startTest();
System.enqueueJob(example);
Test.stopTest();
Assert.areEqual(
2,
[SELECT COUNT() FROM AsyncApexJob WHERE JobType = 'Queueable' AND ApexClass.Name = :QueueableExample.class.getName()]
);
}
}This immediately fails for a reason that might be familiar to you if you’ve spent much time working with Queueables in tests before:
System.AsyncException: Maximum stack depth has been reached.
Class.QueueableExample.execute: line 10, column 1It turns out you can’t re-enqueue the same queueable instance twice within a test. And that’s something of a problem, to be honest: recursive queueables are a common pattern to chunk large data volume on the Salesforce platform, and not being able to test more than one iteration as a result isn’t awesome. Some people have chosen — perhaps rightfully so — to unit test their Queueables by doing something like this:
@IsTest
private class QueueableExampleTest {
@IsTest
static void enqueuesMultipleTimesInTheSameTransaction() {
QueueableExample qex = new QueueableExample();
qex.shouldEnqueueAgain = true;
Test.startTest();
example.execute(null);
Test.stopTest();
Assert.areEqual(
1,
[SELECT COUNT() FROM AsyncApexJob WHERE JobType = 'Queueable' AND ApexClass.Name = :QueueableExample.class.getName()]
);
}
}But that fails for a different reason:
System.HandledException: System.attachFinalizer(Finalizer) is not allowed in this context
Class.QueueableExample.execute: line 5, column 1
Class.QueueableExampleTest.enqueuesMultipleTimesInTheSameTransaction: line 10, column 1This isn’t the end of the world either; we can always do something like:
public class QueueableExample implements System.Queueable {
public Boolean shouldEnqueueAgain = false;
public void execute(System.QueueableContext qc) {
+ if (System.isQueueable()) {
+ System.attachFinalizer(new FinalizerExample());
+ }
- System.attachFinalizer(new FinalizerExample());
// do async stuff here
if (this.shouldEnqueueAgain) {
this.shouldEnqueueAgain = false;
System.enqueueJob(this);
}
}
}This will work, but it’s not realistic. We need to go even deeper to see why.
Going Deeper
The general idea with a recursive queueable isn’t for it to only run twice. In actuality, a much more common pattern is receiving a collection of records and chunking them as needed into an arbitrary number of “batches:”
public class QueueableExample implements System.Queueable {
private Integer chunkSize = 50;
private final List<SObject> records;
public QueueableExample(List<SObject> records) {
this.records = records;
}
public QueueableExample setChunkSize(Integer newChunkSize) {
this.chunkSize = newChunkSize;
return this;
}
public void execute(System.QueueableContext qc) {
Integer counter = 0;
while (records.isEmpty() == false) {
SObject record = records.remove(0);
// do something async with the record
counter++;
if (counter == this.chunkSize) {
break;
}
}
if (this.records.iterator().hasNext()) {
System.enqueueJob(this);
}
}
}
Back in our test, we’re still stuck though:
@IsTest
private class QueueableExampleTest {
@IsTest
static void enqueuesMultipleTimesInTheSameTransaction() {
QueueableExample example = new QueueableExample(
new List<SObject>{ new Account(), new Account(), new Account() }
).setChunkSize(1);
Test.startTest();
System.enqueueJob(example);
Test.stopTest();
Assert.areEqual(
3,
[SELECT COUNT() FROM AsyncApexJob WHERE JobType = 'Queueable' AND ApexClass.Name = :QueueableExample.class.getName()]
);
}
}This is where the finalizer comes in.
public class QueueableExample implements System.Queueable {
private Integer chunkSize = 50;
private final List<SObject> records;
public QueueableExample(List<SObject> records) {
this.records = records;
}
public QueueableExample setChunkSize(Integer newChunkSize) {
this.chunkSize = newChunkSize;
return this;
}
public void execute(System.QueueableContext qc) {
+ System.attachFinalizer(new FinalizerExample(this));
Integer counter = 0;
while (records.isEmpty() == false) {
SObject record = records.remove(0);
// do something async with the record
counter++;
if (counter == this.chunkSize ) {
break;
}
}
- if (this.records.iterator().hasNext()) {
- System.enqueueJob(this);
- }
}
+ public class FinalizerExample implements System.Finalizer {
+ private final QueueableExample example;
+ public FinalizerExample(QueueableExample example) {
+ this.example = example;
+ }
+ public void execute(System.FinalizerContext fc) {
+ if (this.example.records.iterator().hasNext()) {
+ System.enqueueJob(this.example);
+ }
+ }
+ }
}This test passes in 170ms.
Beyond the Basics
This is a huge deal. While there is still a limit to be found here, it’s an outrageously high number of re-enqueuing that the platform allows before eventually throwing:
System.LimitException: Too many async jobs enqueued for this apex test contextFor example, this version of the test passes:
@IsTest
private class QueueableExampleTest {
@IsTest
static void enqueuesMultipleTimesInTheSameTransaction() {
QueueableExample example = new QueueableExample(
new SObject[100]
).setChunkSize(1);
Test.startTest();
System.enqueueJob(example);
Test.stopTest();
Assert.areEqual(
100,
[SELECT COUNT() FROM AsyncApexJob WHERE JobType = 'Queueable' AND ApexClass.Name = :QueueableExample.class.getName()]
);
}
}My point is this — while a single re-enqueing isn’t nearly enough to prove that your bulkified code works, one hundred jobs is more than enough to prove that your code is working, and that’s leaving aside the fact that outside of exceptionally CPU intensive operations you’d likely be using a match higher chunking size.
Promoting State To The Finalizer
There are two issues with the prior implementation:
- the finalizer is tightly coupled to the
QueueableExampleclass since it accesses therecordsvariable; this wouldn’t work if it were an outer class - the finalizer is responsible for deciding whether or not to re-enqueue, which is a responsibility that should really live in the queueable
The nice thing is that both of these issues can be solved neatly:
public class FinalizerExample implements System.Finalizer {
private final System.Queueable example;
private Boolean shouldRestart = false;
public FinalizerExample(System.Queueable example) {
this.example = example;
}
public void execute(System.FinalizerContext fc) {
if (this.shouldRestart) {
System.enqueueJob(this.example);
}
}
}Back in the original QueueableExample, it’s a small change to the logic in execute to support this more generic version:
public void execute(System.QueueableContext qc) {
FinalizerExample finalizer = new FinalizerExample(this);
System.attachFinalizer(finalizer);
Integer counter = 0;
while (this.records.isEmpty() == false) {
SObject record = records.remove(0);
// do something async with the record
counter++;
if (counter == this.chunkSize) {
break;
}
}
finalizer.shouldRestart = this.records.isEmpty() == false;
}
// and just to demonstrate this working, some small tweaks to the test class:
@IsTest
private class QueueableExampleTest {
@IsTest
static void enqueuesMultipleTimesInTheSameTransaction() {
QueueableExample example = new QueueableExample(
new SObject[3]
).setChunkSize(2);
Test.startTest();
System.enqueueJob(example);
Test.stopTest();
Assert.areEqual(
2,
[SELECT COUNT() FROM AsyncApexJob WHERE JobType = 'Queueable' AND ApexClass.Name = :QueueableExample.class.getName()]
);
}
}And just like that, this finalizer implementation works with any queueable.
Layering In The Actual Job Result
Now we have a new problem — an actual exception is occasionally thrown in the QueueableExample, and the fact that the job continues without retrying the record that failed is brought up as a concern by stakeholders. This would be an actual nightmare to resolve without finalizers, since a queueable that’s async can only spawn one additional queueable, and a recursive queueable takes up that one async job slot. Yes, there are other options available, but all of them have tradeoffs. To demonstrate a nicely encapsulated solution, I’ve sprinkled in some debug statements with the updated code to show how object-oriented programming and a finalizer can make things relatively simple:
@IsTest
private class QueueableExampleTest {
@IsTest
static void enqueuesMultipleTimesInTheSameTransaction() {
QueueableExample example = new QueueableExample(
// using actual SObjects now to make what's happening clear
new List<SObject>{ new Account(Name = 'One'), new Account(Name = 'Two'), new Account(Name = 'Three!')}
).setChunkSize(2);
try {
Test.startTest();
System.enqueueJob(example);
Test.stopTest();
} catch (Exception ex) {
// do nothing, the exception is just to show
// that the finalizer is handling the exception properly
}
Assert.areEqual(
2,
[SELECT COUNT() FROM AsyncApexJob WHERE JobType = 'Queueable' AND ApexClass.Name = :QueueableExample.class.getName()]
);
}
}
// and then a slightly modified FinalizerExample
public virtual class FinalizerExample implements System.Finalizer {
public Boolean shouldRestart = false;
protected final System.Queueable example;
public FinalizerExample(System.Queueable example) {
this.example = example;
}
public virtual void execute(System.FinalizerContext fc) {
if (this.shouldRestart) {
System.debug('Starting up again');
System.enqueueJob(this.example);
}
}
}
// and lastly the Queueable
public class QueueableExample implements System.Queueable {
private Integer chunkSize = 50;
private final List<SObject> records;
private final SecondFinalizerExample finalizer = new SecondFinalizerExample(this);
public QueueableExample(List<SObject> records) {
this.records = records;
}
public QueueableExample setChunkSize(Integer newChunkSize) {
this.chunkSize = newChunkSize;
return this;
}
public void execute(System.QueueableContext qc) {
System.debug('Starting up with the following records: ' + this.records);
System.attachFinalizer(finalizer);
Integer counter = 0;
this.finalizer.shouldRestart = true;
while (this.records.isEmpty() == false) {
SObject record = records.remove(0);
this.finalizer.currentRecord = record;
// do something async with the record
counter++;
if (counter == 2) {
throw new IllegalArgumentException('Oops!');
}
if (counter == this.chunkSize) {
break;
}
}
this.finalizer.shouldRestart = this.records.isEmpty() == false;
}
private class SecondFinalizerExample extends FinalizerExample {
private final List<SObject> possibleRetries = new List<SObject>();
public SObject currentRecord;
public SecondFinalizerExample(QueueableExample example) {
super(example);
}
public override void execute(System.FinalizerContext fc) {
System.debug('Possible retries in finalizer: ' + this.possibleRetries);
System.debug('Finalizer result: ' + fc.getResult());
System.debug('Should restart? ' + this.shouldRestart);
if (fc.getResult() == System.ParentJobResult.UNHANDLED_EXCEPTION) {
this.possibleRetries.add(this.currentRecord);
}
super.execute(fc);
if (this.possibleRetries.isEmpty() == false && this.shouldRestart == false) {
QueueableExample typedExample = (QueueableExample) this.example;
typedExample.records.clear();
typedExample.records.addAll(this.possibleRetries);
this.possibleRetries.clear();
System.enqueueJob(this.example);
System.debug('Retrying for failures');
}
}
}
}That produces a set of debug statements like such:
USER_DEBUG|[17]|DEBUG|Starting up with the following records: (Account:{Name=One}, Account:{Name=Two}, Account:{Name=Three!})
USER_DEBUG|[46]|DEBUG|Possible retries in finalizer: ()
USER_DEBUG|[47]|DEBUG|Finalizer result: UNHANDLED_EXCEPTION
USER_DEBUG|[48]|DEBUG|Should restart? true
USER_DEBUG|[12]|DEBUG|Starting up again
USER_DEBUG|[17]|DEBUG|Starting up with the following records: (Account:{Name=Three!})
USER_DEBUG|[46]|DEBUG|Possible retries in finalizer: (Account:{Name=Two})
USER_DEBUG|[47]|DEBUG|Finalizer result: SUCCESS
USER_DEBUG|[48]|DEBUG|Should restart? false
USER_DEBUG|[59]|DEBUG|Retrying for failures
USER_DEBUG|[17]|DEBUG|Starting up with the following records: (Account:{Name=Two})
USER_DEBUG|[46]|DEBUG|Possible retries in finalizer: ()
USER_DEBUG|[47]|DEBUG|Finalizer result: SUCCESS
USER_DEBUG|[48]|DEBUG|Should restart? falseNote that the failed queueable job doesn’t count towards the AsyncApexJob counts. Also note that this is more coupled than the previous example; the QueueableExample needs a subclass for the original finalizer to do extra housekeeping plus the accessing of private variables. On the whole, though, this entire retry mechanism — and the swapping off of which queueable job is responsible for doing what — would have been a lot harder (if not outright impossible) prior to the introduction of finalizers.
Beyond Retries - Integrating Nebula Logger
Now, checking the job result and implementing a retry mechanism is pretty cool, but what we really like this for is creating a one-stop-shop for Nebula Logger to be invoked:
public virtual class FinalizerExample implements System.Finalizer {
public Boolean shouldRestart = false;
protected final System.Queueable example;
public FinalizerExample(System.Queueable example) {
this.example = example;
}
public virtual void execute(System.FinalizerContext fc) {
Logger.setAsyncContext(fc);
Logger.debug('Finalizer result: ' + fc.getResult());
Logger.debug('Should restart? ' + this.shouldRestart);
switch on fc?.getResult() {
when UNHANDLED_EXCEPTION {
this.handleError(fc.getException());
}
}
if (this.shouldRestart) {
System.debug('Starting up again');
System.enqueueJob(this.example);
}
Logger.saveLog();
}
protected virtual void handleError(Exception ex) {
Logger.error('An error occurred during last queueable run').setExceptionDetails(ex);
}
}
// and then in the QueueableExample:
private class SecondFinalizerExample extends FinalizerExample {
private final List<SObject> possibleRetries = new List<SObject>();
public SObject currentRecord;
public SecondFinalizerExample(QueueableExample example) {
super(example);
}
public override void execute(System.FinalizerContext fc) {
Logger.debug('Possible retries in finalizer: ' + this.possibleRetries);
if (this.possibleRetries.isEmpty() == false && this.shouldRestart == false) {
QueueableExample typedExample = (QueueableExample) this.example;
typedExample.records.clear();
typedExample.records.addAll(this.possibleRetries);
this.possibleRetries.clear();
System.enqueueJob(this.example);
Logger.debug('Retrying for failures');
}
super.execute(fc);
}
protected override void handleError(Exception ex) {
this.possibleRetries.add(this.currentRecord);
super.handleError(ex);
}
}This considerably simplifies what would have required individual try/catch blocks at any sensitive point in existing queueables; it standardizes the process by which queueables can be considered production-ready by automatically tracking failures and allowing you to subclass out custom functionality as required. Indeed, since now we can take advantage of awesome methods like Logger.setAsyncContext, we automatically get more nuanced info about our job:
USER_DEBUG|[3476]|INFO|Nebula Logger - Async Context: {
"type" : "System.FinalizerContext",
"parentJobId" : "7076g0000B6tiBZAQY",
"finalizerResult" : "UNHANDLED_EXCEPTION",
"finalizerException" : "Oops!\nExternal entry point"
}You might choose to only save logs when an unhandled exception is encountered within a finalizer. The nice thing is that you get to make choices that then ripple out and have an immediately positive effect throughout your codebase; because adding a finalizer is a one-liner, it’s easy to enforce that new and existing Queueables all make use of something like this logging finalizer.
Recovering From Uncommitted Work Errors
Another area where finalizers are extremely useful are scenarios where you have to (and in some cases, are forced to) perform DML between callouts. This is another one of those nightmare situations that previously would have required hacking to really get the desired effect, but it’s not out of the question to need to perform DML between two callouts (and, for some examples like certain Connect API methods, DML will be performed behind the scenes anyway). We had this experience recently while working with the Apex Slack SDK, which adds another limit-based paradigm on top of everything else:
- the Apex Slack SDK requires Slack-initiated actions to return a response within 3 seconds, which (for any complicated processing) requires async work
- the first callout would occur in a queueable
- there was pre-existing DML being performed after that
- a new requirement came up for a second callout to be performed which relied on data being created post-DML
Finalizers enable this use-case without a huge lift. At this point you’ve probably seen enough in the way of examples to see what I’m talking about, but I’m hopeful by including this info it gets you to thinking: what couldn’t I do before that finalizers allow me to do?
Wrapping Up
The platform continues to improve upon Queueable finalizers. For example, if you called System.attachFinalizer within a queueable that had already attached a finalizer, the platform used to throw an exception. Now that’s handled for you. It’s my belief that the addition of finalizers has opened up entirely new modalities within the already vast realm of asynchronous programming within Apex, and this post is just brushing the surface as far as what using Finalizers allows you to do.
As always, a huge thank you for reading the Joys Of Apex, and a shoutout to Arc and Henry for sponsoring me on Patreon. Let me know how you’re using finalizers, or something you learned while reading this that you’re excited to try out. Till next time!