Two Years In Open Source

Last year, I wrote about A Year In Open Source, and I wanted to wrap this year up with another reflection after having hundreds of hours over the course of 2022 engaged in giving back. I’ve had the pleasure of mentoring colleagues & friends; of contributing additions to popular open source libraries like Nebula Logger, Trigger Action Framework as well as my own repositories; of reading and continuing to hone my own skills. I was also a guest speaker at an internal symposium for Open Source we had here at Salesforce, which was a real pleasure. Join me for a look back on the successes and failures of the last year.

Light Reading (Continued)

One of the primary challenges that we face as we continue to progress in our software development careers lies in finding new material to learn from and to inspire, particularly of the sort that I think serves as a force-multiplier to developers — advanced guides to concepts that can be applied within any language. With that in mind, there are really only two articles I’ve come across this year that serve to demonstrate the kind of thought development I’m talking about:

• Abstraction: Not What You Think It Is
• The Best Refactoring You've Never Heard Of

I tend to be skeptical of blogs that serve advertisements, as this one does, but the pure substance in each of these articles and the food for thought they’ve provided for me over the past year have been immeasurable, and I was willing to forgive the author for a bit of unbridled capitalism as a result. I’d highly recommend reading the articles in their entirety. Like all good long-form writing, these are not necessarily the sort of thing you should expect to read in a single sitting.

Miscellaneous Open Source Work

This year I contributed three big PRs (and reviewed six!) to Nebula Logger:

• Big Object Plugin — completely insane that this was merged this year, as it feels like a lifetime ago since I started work on it (primarily due to the fact that I did indeed start the work more than two years ago!). Added support for Salesforce’s NoSQL solution, also known as Big Objects. Because NoSQL records are, essentially, documents typically only easily accessible by a primary key, they come with lots of “fun” edge cases that can really … cut you. Probably our favorite one was “records inserted in a test context aren’t rolled back and are actually created in your database.”
• Async Error Logging Plugin — starting to realize I worked on a lot of plugins this year … this was a really nice use-case for code reuse, given the two dominant forms of async threading on the Salesforce platform have disparate error conditions and error handling, and unifying that handling for a logging solution is incredibly satisfying. My team uses this plugin at work (in conjunction with Nebula’s ability to send messages to Slack), which means that we get nice, really granular error messages even if we experience an unhandled fault.
• Flow Execution Error Event Plugin — a continuation of the previous plugin, this time including logging support for unhandled faults experienced using Salesforce’s predominant GUI-based automation tool. Not every part of the GUI tool supports the emitting (and thus handling) of these error events, which is an area I’m eager to see improved upon in future releases

As well, I contributed a small PR to the Trigger Action Framework, which is a great example of the Replace Conditional With Polymorphism refactoring technique.

In general I found it hard to find the time to contribute to other projects given the continued work on Apex Rollup, though as of late that’s begun to slow down — which has been a phenomenal, if bittersweet, feeling. With (as of writing) exactly 150 production Salesforce instances using the package, this year saw triple digit adoption growth rate and some really interesting problems to solve. I say “bittersweet” because it’s been an incredibly rewarding two years, and now that the general feature and bug fixing work has trended downwards, the “big ticket” items are harder and harder to come by. For those unfamiliar, Apex Rollup sits on top of standard Salesforce architecture, and augments the platform’s built-in “roll up” summary capabilities (similar to the syntax of the same name in SQL) by quite a bit. The tool, in addition to offering more in the way of supported operations (more on this later, in the Imagining New Functionality section), also adds capabilities like rolling up through several foreign key relationships at once; doing things like summing item A with a key to B (which has keys to C) all the way to C without the use of intermediate fields. For those whose familiarity lies more with SQL than the Salesforce query langauge (SOQL), rollup fields are primarily exciting for their ability to group “child” data (multiple records, or rows, in any given table with matching foreign keys to a row in another table) at the “parent” level. While there exists limited support for a small group of aggregated functions — think COUNT, SUM, MIN and MAX — out of the box, it takes a custom solution to provide more more advanced and nuanced rollup functionality (things like AVERAGE, CONCAT, FIRST, LAST, etc …).

I think that’s a nice segue into some interesting architectural events that have occurred over the past year.

Architectural Choices Over The Last Year

Advanced Support For Multiple Currencies

One of the big contributions that Jonathan made for Apex Rollup was the addition of multi-currency support back in 2021. This means that for currency-based fields, in Salesforce instances where multiple currencies are enabled, Apex Rollup can convert those fields into the targeted parent field’s currency when doing calculations. However, that’s only baseline multiple currency support. More advanced Salesforce instances can have dated exchange rates, which more closely align to real world conditions and can accurately account for things like exchange rate fluctation over time. Since we’ve seen quite a bit of exchange rate fluctation over the past two years, it was somewhat surprising to have it go till now prior to the more advanced multicurrency features being requested. Here’s what the initial shape of the prior version of multicurrency (without dated exchange rate support) looked like:

public class RollupCurrencyInfo {
  private static final Map<String, RollupCurrencyInfo> CURRENCY_ISO_CODE_TO_CURRENCY {
    get {
      if (CURRENCY_ISO_CODE_TO_CURRENCY == null) {
        CURRENCY_ISO_CODE_TO_CURRENCY = getCurrencyMap();
      }
      return CURRENCY_ISO_CODE_TO_CURRENCY;
    }
    set;
  }
  private static final Boolean IS_MULTICURRENCY = UserInfo.isMultiCurrencyOrganization();

  public String IsoCode { get; set; }
  public Decimal ConversionRate { get; set; }
  public Integer DecimalPlaces { get; set; }

  private static Map<String, RollupCurrencyInfo> getCurrencyMap() {
    // some test-specific stuff, followed by ...
    // first hurdle - not all Salesforce instances have multicurrency enabled
    // so there's a guard clause here to prevent trying to query a table that
    // won't exist otherwise. Since the "shape" of the data corresponds
    // to the public fields above, with a little hacky serialize/deserialize
    // trip we can safely access the data without declaring a strongly-typed version
    // of the underlying table (normally preferable)
    // but here would lock us into only being able to install our build artifict into Salesforce
    // orgs where multicurrency is enabled.
    Map<String, RollupCurrencyInfo> currencyInfoMap = new Map<String, RollupCurrencyInfo>();
    if (IS_MULTICURRENCY == false) {
      return currencyInfoMap;
    }

    String query = 'SELECT IsoCode, ConversionRate, DecimalPlaces, IsCorporate FROM CurrencyType WHERE IsActive = TRUE';
    List<RollupCurrencyInfo> currencyTypes = (List<RollupCurrencyInfo>) JSON.deserialize(JSON.serialize(Database.query(query)), List<RollupCurrencyInfo>.class);
    for (RollupCurrencyInfo currencyType : currencyTypes) {
      currencyInfoMap.put(currencyType.IsoCode, currencyType);
    }
    return currencyInfoMap;
  }

  // some other static methods for converting children currency amounts
  // to their respective parent currency amounts
}

It’s a relatively simple class; the majority of the initial implementation was simply plugging this into the larger architecture. I knew that eventually advanced multicurrency support would be requested, and both Jonathan and myself probably could have optimized for that with the initial release. So what should one do, knowing a thing like this?

At the time the PR came through, I could have chosen to do a few things:

• done the discovery work up-front to see what, if anything, differed for the date range versions of currency information
• delayed the release of the initial feature (itself, something that had been requested) till the larger functionality was feature-complete
• create stubs for the advanced date ranges, but otherwise leave it unimplemented till requested

I chose a fourth option: doing nothing. It’s something that bugged me, every now and then, in the meantime (leaving feature work undone?? Who does that?!) but since it took more than a year for the advanced multicurrency support to be requested, I’ve had ample time to reflect on what a good decision that was. Doing nothing meant getting back to other pressing feature requests faster; it meant being able to continue to quickly respond to change.

So that’s the first takeaway — don’t build things that don’t provide value. Don’t overarchitect when less will do.

When it came time to actually update the class shown to provide support for dated exchange rates, I had some hesitation over updating the existing dictionary struct (Map<String, RollupCurrencyInfo>) to something one-to-many in nature, given that the majority of use-cases would continue to only contain a mapping between a single currency code to exchange rate. Because the shape of the underlying data was incredibly similar (with the additiion of bounding date fields for each dated exchange rate), this was a perfect opportunity to use the Proxy pattern, as it helped to minimize the number of changes made.

First off was making the base RollupCurrencyInfo class virtual, and swapping out where in the implementation those classes were being used by making the class itself into a proxy:

public virtual class RollupCurrencyInfo {
  // properties, and the static map you saw earlier

  // the proxy method
  protected virtual RollupCurrencyInfo getInfo(SObject calcItem, String isoCode) {
    return this;
  }

  // a no-op by default
  protected virtual void addInfo(RollupCurrencyInfo info) {
  }

  // this is the old version of the code:
  // Decimal calcItemAmountInOrgCurrency = CURRENCY_ISO_CODE_TO_CURRENCY.get(calcItemIsoCode).ConversionRate / calcItemDenominator;
  // Decimal calcItemAmountInParentCurrency = CURRENCY_ISO_CODE_TO_CURRENCY.get(parentIsoCode).ConversionRate / calcItemAmountInOrgCurrency;
  // and the updated version of it, now calling the proxy:
  Decimal calcItemAmountInOrgCurrency =
        CURRENCY_ISO_CODE_TO_CURRENCY.get(calcItemIsoCode).getInfo(calcItem, calcItemIsoCode).ConversionRate / calcItemDenominator;
  // changed usage type here to Double to deal with some scientific notation number formatting
  Double calcItemAmountInParentCurrency =
      (CURRENCY_ISO_CODE_TO_CURRENCY.get(parentIsoCode).getInfo(calcItem, parentIsoCode).ConversionRate / calcItemAmountInOrgCurrency)
      .doubleValue();
}

With the base class successfully proxied, the only thing left to do was provide the updated implementation when dated exchange rates were being used and supply that as the RollupCurrencyInfo instance being keyed to each currency when applicable:

// in the "getCurrencyInfo" method shown earlier, using our second proxied method for "addInfo"
for (RollupCurrencyInfo currencyInfo : currencyInfos) {
  if (currencyInfoMap.containsKey(currencyInfo.IsoCode)) {
    currencyInfoMap.get(currencyInfo.IsoCode).addInfo(currencyInfo);
  } else {
    RollupCurrencyInfo mappedInfo = IS_DATED_MULTICURRENCY ? new CurrencyFinder(currencyInfo) : currencyInfo;
    currencyInfoMap.put(currencyInfo.IsoCode, mappedInfo);
  }
}
return currencyInfoMap;

private class CurrencyFinder extends RollupCurrencyInfo {
  private final List<RollupCurrencyInfo> currencyInfos = new List<RollupCurrencyInfo>();
  private final Map<Id, RollupCurrencyInfo> cachedItemToInfo = new Map<Id, RollupCurrencyInfo>();
  private RollupCurrencyInfo baseInfo;

  public CurrencyFinder(RollupCurrencyInfo info) {
    this.addInfo(info);
  }

  protected override void addInfo(RollupCurrencyInfo info) {
    if (info.StartDate != null) {
      this.currencyInfos.add(info);
    } else {
      this.baseInfo = info;
    }
  }

  protected override RollupCurrencyInfo getInfo(SObject calcItem, String isoCode) {
    if (cachedItemToInfo.containsKey(calcItem.Id)) {
      return cachedItemToInfo.get(calcItem.Id);
    }
    // unshown helper for finding the Date instance that currencies are keyed off of
    Date currencyDate = getCurrencyDate(calcItem);
    if (this.currencyInfos.isEmpty() == false && currencyDate != null) {
      for (RollupCurrencyInfo info : this.currencyInfos) {
        if (info.IsoCode == isoCode && info.NextStartDate > currencyDate && currencyDate >= info.StartDate) {
          cachedItemToInfo.put(calcItem.Id, info);
          return info;
        }
      }
    }
    cachedItemToInfo.put(calcItem.Id, this.baseInfo);
    return this.baseInfo;
  }
}

Et voilà! With just a small bit of logic in RollupCurrencyInfo.CurrencyFinder to seek the proper bit of info using a sentinel currencyDate supplied by each record, and with the help of our proxied base class, the existing implementation was kept largely unchanged, with only the net-new functionality added. I recently described refactoring to patterns as:

building up a vocabulary of higher-level abstractions … the more “words” you have in your vocabulary, the more likely it is you can look at something and have an idea of how to refactor it.

Being able to identify the importance of being able to turn something that could be undesirable (allocating to the heap unnecessarily with a one-to-many dictionary being a prime example) into something desirable (adding requested functionality with only a few small tweaks to the existing code) is the second takeaway. That’s the power of patterns.

Imagining New Functionality

One of the features that I’ve been most excited about … rolling out … this past year were some new kinds of rollups. A long time ago (March 8th, 2021, if we’re being exact), somebody had asked me about whether or not I supported rolling up Boolean-based values using Apex Rollup. This was my response:

It’s funny the sort of fruit that one can grow out of such small seeds. It took me a while, but I routinely thought back to that conversation — I wanted to be sure I wasn’t missing out on some opportunity to create helpful functionality, and the answer came to me one day while my wife and I were waiting for a flight. I had been doing some C# work, and was thinking about an old bug in an application I’d worked on caused by one of the predicate-based filters within Linq — maybe it was .Any() or .All().

For those unfamiliar with C#, these are collection-filtering methods that, via generic overloads, allow you to supply a lambda function for filtering:

using System;
using System.Linq;
using System.Collections.Generic;

class Program {
  static void Main(string[] args) {
    Console.WriteLine(hasAnyEvens(new List<int>(){ 1, 3, 3 })); // False
    Console.WriteLine(hasAnyEvens(new List<int>(){ 1, 2, 3 })); // True
  }

  static bool hasAnyEvens(IEnumerable<int> nums) => nums.Any(num => num % 2 == 0);
}

So I was thinking about those methods, and about the whole “what should a parent’s field end up reflecting if two out of four booleans are true?” question, and suddenly, inspiration struck — why not let people define which version of the operation they cared about via a where clause, just like Linq allows? The codebase already had the two primary prerequisites met:

• ability to define conditional filters
• ability to define rollup calculators based on the presence of a guiding enum

Creating ALL, SOME and NONE operations wasn’t going to require much new code. I started to get excited about this approach. The entirety of the implementation came in a small inner class:

private without sharing class ConditionalCalculator extends RollupCalculator {
    // constructor omitted passing some data to the parent class
    // including the "where clause" which is used in the "winnowItems"
    // function below

    public override void performRollup(List<SObject> calcItems, Map<Id, SObject> oldCalcItems) {
      this.returnVal = null;
      // winnowItems modifies the list passed in by reference
      //  so we clone it here. it also short-circuits and returns early
      // if the operation is SOME and it finds a match
      List<SObject> filteredItems = this.winnowItems(new List<SObject>(calcItems), oldCalcItems);
      Boolean matches = false;
      switch on this.op {
        when ALL {
          matches = calcItems.size() == filteredItems.size();
        }
        when NONE, SOME {
          matches = filteredItems.isEmpty() == this.op == Rollup.Op.SOME == false;
        }
      }

      // configuration can dictate a "default" value
      // which for these mean an early exit for non-matches
      if (this.defaultVal != null && matches == false) {
        this.returnVal = this.defaultVal;
        return;
      }

      switch on this.opFieldOnLookupObject.getDescribe().getType() {
        when CURRENCY, DOUBLE, INTEGER {
          this.returnVal = matches ? 1 : 0;
        }
        when STRING, TEXTAREA {
          this.returnVal = String.valueOf(matches);
        }
        when else {
          this.returnVal = matches;
        }
      }
    }
  }

In looking back on the past year, which also saw the addition of the MOST rollup operation with an even smaller implementation — allowing users to roll up the most commonly occurring value from a set of children — it’s nice to see small, granular changes that don’t edit the existing codebase so much as make meaningful additions to it. An area where I can definitely improve upon is better documenting this functionality. While in the act of writing this section, I fielded a question from a user about this very set of rollup operations because they hadn’t upgraded in a while and were unaware this functionality had been added. To that end, I’m hoping to record a set of videos documenting the setup for all of the different rollup operations.

Coping With Regressions

Of course, no code is perfect, and despite being an avid TDD practitioner, there are plenty of opportunities for me to introduce regressions; for the “unknown unknowns” to leap out and bite. Here’s a breakdown of a recent snapshot of the repository:

Suffice it to say, there’s ample room for me to have messed up, though I’m frequently cutting lines of code overall out of the existing codebase (either by promoting code-driven behavior to configuration, or by identifying unintended duplication). I’ve seen a few instances this year where the error was actually in the way I’d set up a test; to that end, I recently wrote a few Powershell scripts that allow for large data volume (LDV) testing to be performed without being constrained by the sort of limits that are present in true Apex unit tests. This allowed me to find and fix an issue that had been unknowingly introduced some time ago, but hadn’t been reported because the rollups being calculated would have had to have contained over 50,000 children records (much to my chagrin).

I still believe in the guiding power that TDD can allow us to exert over architecture by driving the design of a program organically over time, and there’s still nothing better than pulling off a big refactor with all of the tests passing. As such, any regressions that are reported to me end up codified in additional unit tests (or, failing that, integration tests, now that I’ve created a framework for making these easily); over time this means that the surface area for potential problems diminishes in size.

Earlier this year, I worked with a user who was several versions behind. They’ve elevated how I work with consumers of the Apex Rollup package by having tons of unit tests of their own surrounding the rollups they’ve configured with the package. Some of their unit tests were failing when trying to upgrade the package. This turned into a process that took over a month to get resolved — as it turned out, the old functionality had worked more by happy coincidence than on purpose, and the updates I’d made revealed that some of the ensuing work I’d done on the query engine for Apex Rollup weren’t quite thorough enough.

As the weeks wore on with the problem still unresolved, there were times where I was tempted to sweep the whole thing under the rug by reverting the query engine updates. I couldn’t do that, obviously; though I’d exposed a flaw in some of the logic I’d introduced, I’d also patched several issues for a few other people. In general, quality was veering upwards; going back wouldn’t have only been accepting defeat, it would have been settling for flawed behavior. Here’s the most relevant lines in question from the initial update:

for (SObjectType calcType : typeToWrappedMeta.keySet()) {
  RollupMetadata wrappedMeta = typeToWrappedMeta.get(calcType);
+  wrappedMeta.whereClause = wrappedMeta.concatenateWhereClauses();
-  wrappedMeta.whereClause = wrappedMeta.fullRecalcMeta.whereClause;
  String queryString = RollupQueryBuilder.Current.getQuery(
    calcType,
    new List<String>(wrappedMeta.queryFields),
    'Id',
    '!=',
    wrappedMeta.whereClause
  );
  RollupFullRecalcProcessor fullRecalc = buildFullRecalcRollup(wrappedMeta, queryString, calcType, localInvokePoint);
  # etc ...
}

Without getting too much into the weeds on this one, hopefully the diff illustrates how the old version would only use the very last where clause supplied; fine, if only one operation was being recalculated at a time, or if all of the recalculations had no where clauses, but definitively not fine if some of the recalculations being performed had mutually exclusive where clauses. After a few complicated iterations that ironed out edge cases in how concatened query strings should be formatted, the issue was finally resolved.

Lastly, there were some funny “misses” in terms of functionality along the way. I’ve had a lot of fun fleshing out the plugin infrastructure, which allows for more complicated functionality to be built on top of the Apex Rollup framework without bloating the already large core codebase. There are two logging plugins, for example:

• one that uses a lightweight logger to persist logging messages to the database
• an adapter plugin that sends rollup logging to Nebula Logger

Both of these logging plugins, as well as a built-in logger that prints out messages to the debug log console, rely on a configuration flag. Consumers can also define their own logging implementations, so long as they conform to this interface:

// in general, not a huge fan of prepending interfaces
// with the letter "I", but Apex Langauge Server tooling
// isn't always working and at least this way the type
// signature is advertising "what it is" for the world to see
public interface ILogger {
  void log(String logString, LoggingLevel logLevel);
  void log(String logString, Object logObject, LoggingLevel logLevel);
  void save();
}

When logging isn’t enabled via configuration, there should just be a single message printed to the debug console telling anybody looking at the raw logs that logging is disabled:

USER_DEBUG|WARN|Rollup v1.5.38-beta: logging isn't enabled, no further output
USER_DEBUG Class.Rollup.performRollup: line 698, column 1

The only problem? The orchestration layer for logging was missing that check when seeing whether or not logging data should be persisted:

// the logging mediator
private class CombinedLogger implements ILogger {
  private final List<ILogger> loggers;
  // ...

  public void save() {
    // oops, there wasn't a guard clause here
    for (ILogger logger : this.loggers) {
      logger.save();
    }
  }
}

So even for consumers that had disabled logging, they were getting one message persisted saying that logging wasn’t enabled 🤦‍♂️! Adding a guard clause there to only save when logging was enabled was an easy fix; it was also an unfortunate oversight that had remained for years. Such is life. The code isn’t perfect, today, but I’m continually working with users to improve functionality, setup, and configuration as issues arise.

Questions From The Salesforce Open Source Conference

At the internal conference for open source we had here at Salesforce, one of the questions I was asked during the Q & A section of my own presentation was “how do you find so much time to contribute to open source work?” and I think the response bears repeating here.

I talked about not having kids, and having a dog that wakes up early. That’s certainly part of it. The vast majority of my open source work occurs between the hours of 6 AM - 8 AM, for better or for worse. Still, I think it’s also nice to have a passion project, outside of work, and to use that project as a vehicle for giving back. In a world where so many of our interactions end up transactional, I want to renege on the “hustle culture” ideology. Not everything I do has to make money. As I discussed in Building An Apex Portfolio, if the side-effect of giving back also happens to show what I’m capable of, so be it. It’s far better (in my eyes) than serving ads, or trying to sell you on some subscription service (which I don’t have). I did recently create a Patreon account (shoutout to Henry Vu for his continued support there!) but otherwise am perfectly content with being able to write my thoughts here for the low low cost of nothing.

Another question that I got was “how do I get started contributing to open source?” — my recommended approach is to search GitHub for your language of choice, and then go through some of the top-starred repositories. I know I’ve spent a lot of time in prior posts talking about why I think being able to read code is so important, and how flexing that “muscle” will always be good. Going through top-starred repositories and reading the code in them is good for two reasons:

• it exposes you to potentially idiomatic practices in your language of choice
• it gives you the chance to vet whether or not you agree with architectural, stylistic, and implementation details in a constructive way (where the end result is known)

These things may give you the confidence to start a project of your own, or you might find an existing project that you really love with room to contribute to. Either way, you’ll have learned something valuable.

Here’s an example, found totally at random, while searching for “Apex” on GitHub:

// in some class

protected virtual void clearFields(Set<SObjectField> fields)
{
  for (SObject record : getRecords())
  {
    for (SObjectField field : fields)
    {
      record.put(field, null);
    }
  }
}

// ... many more methods

protected virtual List<SObject> getRecordsByFieldValues(SObjectField field, Set<Object> values)
	{
		List<SObject> result = new List<SObject>();
		for (SObject record : getRecords())
		{
			if (values?.contains(record.get(field)))
			{
				result.add(record);
			}
		}
		return result;
	}

// ... many more methods

protected virtual void setFieldValue(SObjectField field, Object value)
{
  for (SObject record : getRecords())
  {
    record.put(field, value);
  }
}

There are a couple of things that are interesting here, of which I’ll pluck two examples out (and I never would have seen these particular examples if I hadn’t been writing this article!):

• first, in getRecordsByFieldValues - the line if (values?.contains(record.get(field))) — if the values collection actually is null, this if statement would throw because if(null) isn’t valid runtime syntax (you have to coerce to a Boolean value, using something like if (values?.contains(record.get(field))) == true)
• secondly, why isn’t clearFields calling setFieldValue ? It could be refactored easily to:

// trying to preserve the original formatting ...
protected virtual void clearFields(Set<SObjectField> fields)
{
  for (SObjectField field : fields)
  {
    this.setFieldValue(field, null);
  }
}

I also found it interesting while looking at so many other overloaded methods in this same class that there wasn’t an overloaded version of setFieldValue to prevent having to loop over the records many times. These are the sorts of interesting things you can run across and judge for yourself when getting involved with open source — you’re always just one pull request away from being able to improve things! Again, worst case scenario, you learn something along the way like “is it a code smell if all of these functions are iterating over the same collection and none of them have been generalized at all to prevent that?”

Wrapping Up

It’s been another formative year spent writing code & articles inspired by that code. As we head into 2023, I’m excited to continue to work on Apex Rollup and other open source endeavors. I hope this reflection piece was an enjoyable one. It was good to look back on the good & bad that came out of my own open source work this year; hopefully it’s been helpful sharing both sides of the journey. Here’s wishing you all the best as we close out this year and head into the new one! As well, don’t miss out on 2023’s reflection piece: Three Years In Open Source.