Benchmarking Matters
Table of Contents:
It’s never too soon (or too late) to test our understanding of programming fundamentals. In this post, I want to explore two very simple benchmarking tests that show how small platform changes can lead to large speed improvements. First, I’ll walk you through some code you might remember from Sorting & Performance In Apex to show off how the existing enhancements made to sorting in Winter ‘24 have led to major sorting performance enhancements. Secondly, we’ll take a look at the upcoming support for null coalescing in Spring ‘24, and how that’s already showing strong performance improvements over the branching conditional equivalents.
Creating Ranges In Apex
But first! A short while ago, I wrote on LinkedIn about one of the other excellent enhancements to Apex within Winter ‘24:
Iterate Within For Loops More Easily with Iterable - You can now easily iterate through lists or sets using an Iterable variable in a for loop.
In order to aid in our benchmarking, I’d like to easily be able to define a range to iterate over. Back in the day, this would have required a vanilla for loop:
private static final Integer MAX_ITERATION_AMOUNT = 1000;
for (Integer index = 0; index < MAX_ITERATION_AMOUNT; index++) {
// benchmarking stuff here
}That’s nice and clear and repetitive. Instead, I’d love to write:
for (Integer index : Range.create(ITERATION_MAX)) {
// benchmarking stuff here
}In Winter ‘24, that’s easy:
public class Range implements System.Iterable<Integer> {
private final List<Integer> ints = new List<Integer>();
private Range(Integer starting, Integer ending) {
for (Integer index = starting; index < ending; index++) {
this.ints.add(index);
}
}
public static Range create(Integer startingNumber, Integer endingNumber) {
return new Range(startingNumber, endingNumber);
}
public static Range create(Integer endingNumber) {
return new Range(0, endingNumber);
}
public System.Iterator<Integer> iterator() {
return this.ints.iterator();
}
public Integer size() {
return this.ints.size();
}
}I can easily demonstrate with tests that Range does all the things I need it to do, performantly:
@IsTest
private class RangeTests {
@IsTest
static void sizeReflectRangeAmount() {
Assert.areEqual(50, Range.create(50, 100).size());
Assert.areEqual(1, Range.create(0, 1).size());
}
@IsTest
static void iteratesThroughRange() {
Integer startingCount = 0;
for (Integer intermediate : Range.create(startingCount, 50)) {
Assert.areEqual(startingCount, intermediate);
startingCount++;
}
Assert.areEqual(50, startingCount);
}
@IsTest
static void iteratesEfficientlyOverLargeNumbers() {
Long nowInMs = System.now().getTime();
for (Integer unused : Range.create(100000));
Assert.isTrue(System.now().getTime() - nowInMs <= 1000);
}
}Benchmarking Sorting Implementations
Now I can easily create some benchmarking tests:
@IsTest
private class BenchmarkingTests {
private static final Integer MAX_ITERATION_AMOUNT = 1000;
@IsTest
static void benchmarksOldCustomSorter() {
List<Account> unsortedAccounts = new List<Account>();
for (Integer index : Range.create(10)) {
unsortedAccounts.add(new Account(Industry = '' + index + ' unsorted'));
}
OldComparator oldSorter = new OldIndustrySorter();
for (Integer index : Range.create(ITERATION_MAX)) {
oldSorter.sort(unsortedAccounts);
}
}
}I’ll bring back the Comparator class shown off in Sorting & Performance In Apex:
// within BenchmarkingTests:
public abstract class OldComparator {
public abstract Integer compare(Object o1, Object o2);
public void sort(Object[] values) {
ItemWrapper[] wrappedItems = new List<ItemWrapper>();
for(Object value: values) {
wrappedItems.add(new ItemWrapper(this, value));
}
wrappedItems.sort();
values.clear();
for(ItemWrapper item: wrappedItems) {
values.add(item.value);
}
}
}
private class ItemWrapper implements System.Comparable {
private final OldComparator comparer;
private final Object value;
public ItemWrapper(OldComparator comparer, Object value) {
this.comparer = comparer;
this.value = value;
}
public Integer compareTo(Object o) {
return this.comparer.compare(value, ((ItemWrapper)o).value);
}
}Then I’ll write a test for the new sort(System.Comparator<T> customSorter) implementation introduced in Winter ‘24:
@IsTest
static void benchmarksWinterTwentyFourSorting() {
List<Account> unsortedAccounts = new List<Account>();
for (Integer index : Range.create(10)) {
unsortedAccounts.add(new Account(Industry = '' + index + ' unsorted'));
}
System.Comparator<Account> sorter = new WinterTwentyFourIndustrySorter();
for (Integer index : Range.create(ITERATION_MAX)) {
unsortedAccounts.sort(sorter);
}
}
private class WinterTwentyFourIndustrySorter implements System.Comparator<Account> {
public Integer compare(Account one, Account two) {
return one.Industry.compareTo(two.Industry);
}
}The results are shocking (with ITERATION_MAX set to 1000):
=== Test Results
TEST NAME OUTCOME MESSAGE RUNTIME (MS)
─────────────────────────────────────────────────── ─────── ─────── ────────────
BenchmarkingTests.benchmarksOldCustomSorter Pass 569
BenchmarkingTests.benchmarksWinterTwentyFourSorting Pass 165That’s a 110% improvement in speed from the old to the new!
Interestingly, for larger values of ITERATION_MAX, the newer System.Comparator<T> interface continues to shine; at 100000 iterations, the old implementation starts to fail completely:
=== Test Results
TEST NAME OUTCOME MESSAGE RUNTIME (MS)
─────────────────────────────────────────────────── ─────── ──────────────────────────────────────────────────────────────────── ────────────
BenchmarkingTests.benchmarksOldCustomSorter Fail System.LimitException: Apex CPU time limit exceeded
Class.BenchmarkingTests.OldComparator.sort: line 67, column 1
Class.BenchmarkingTests.benchmarksOldCustomSorter: line 30, column 1
BenchmarkingTests.benchmarksWinterTwentyFourSorting Pass 14705The TL;DR here is that if you’re using an old custom sort implementation, particularly for areas of your codebase where large data volume updates can be expected, you can significantly speed up performance by porting your sorting (😇) to System.Comparator<T>.
Benchmarking Null Coalesce Versus Conditionals
These tests are even simpler, to begin with:
@IsTest
static void benchmarksIfStatement() {
for (Integer potentiallyNull : Range.create(ITERATION_MAX)) {
Integer assignment = potentiallyNull;
if (assignment == null) {
assignment = 0;
}
}
}
@IsTest
static void benchmarksNullCoalesceStatement() {
for (Integer potentiallyNull : Range.create(ITERATION_MAX)) {
Integer assignment = potentiallyNull ?? 0;
}
}It’s already possible to show how much more performant null coalescing is:
=== Test Results
TEST NAME OUTCOME MESSAGE RUNTIME (MS)
─────────────────────────────────────────────────── ─────── ─────── ────────────
BenchmarkingTests.benchmarksIfStatement Pass 56
BenchmarkingTests.benchmarksNullCoalesceStatement Pass 16Let’s look at a more complicated example that better demonstrates how null coalescing in conjunction with safe navigation can improve complex pieces of code:
private static final List<Account> FIXED_ACCOUNTS = new List<Account>{
null, new Account(), null, new Account(Name = 'Hello!')
};
@IsTest
static void benchmarksComplexIfStatement() {
List<String> accountNames = new List<String>();
for (Integer index : Range.create(ITERATION_MAX)) {
for (Account acc : FIXED_ACCOUNTS) {
String accountName = acc?.Name;
if (accountName == null) {
accountName = 'Fallback';
}
accountNames.add(accountName);
}
}
}
@IsTest
static void benchmarksComplexNullCoalesce() {
List<String> accountNames = new List<String>();
for (Integer index : Range.create(ITERATION_MAX)) {
for (Account acc : FIXED_ACCOUNTS) {
accountNames.add(acc?.Name ?? 'Fallback');
}
}
}If you’re curious, after the first iteration for the second test, the accountNames list looks like this:
(Fallback, Fallback, Fallback, Hello!)And we can see that null coalescing is still introducing some nice performance improvements (again with ITERATION_MAX set to 1000):
=== Test Results
TEST NAME OUTCOME MESSAGE RUNTIME (MS)
─────────────────────────────────────────────── ─────── ─────── ────────────
BenchmarkingTests.benchmarksComplexIfStatement Pass 106
BenchmarkingTests.benchmarksComplexNullCoalesce Pass 61If you’re curious, for larger values of ITERATION_MAX like 100000, there’s a very similar performance difference — but we really start to see how inefficient nested for loops can be, as well:
=== Test Results
TEST NAME OUTCOME MESSAGE RUNTIME (MS)
─────────────────────────────────────────────── ─────── ─────── ────────────
BenchmarkingTests.benchmarksComplexIfStatement Pass 40104
BenchmarkingTests.benchmarksComplexNullCoalesce Pass 22142Yikes! A good lesson to us all to be mindful of nested iterations, even when testing.
Using Tools For Benchmarking
Apex Unit tests are and should be your first tool and line of defense when it comes to benchmarking. Sometimes, though, it’s necessary to dive deep down into the code to understand the performance of individual methods in a way that the overall testing framework can’t quite help with. For times like this, I highly recommend the Apex Log Analyzer, which allows you to introspect on debug log data to do things like:
- see the individual performance of methods
- measure how long actions like DML/invocables/SOQL take with respect to the entire transaction
- aggregate method usage and how it contributes to overall transaction time
I cannot recommend this tool enough! When investigating performance issues, or just benchmarking performance in general, it’s completely invaluable.
If you’re interested in benchmarking, check out the highlights within Four Years In Open Source, as one of them is a contribution to Nebula Logger that reduced logging time anywhere between 50-80%, and I used the log analyzer to achieve those results!
Wrapping Up
If you’re interested in browsing the source code for this post directly, you can do so on GitHub. I hope this opens your eyes to some low-hanging fruit when it comes to optimizing your Apex code’s performance.
As always, thanks to Henry Vu and the rest of my sponsors on Patreon. Your continued support and encouragement means a lot to me!