A Year In Open Source

At the end of 2020, I decided to take 2 weeks off of work at the end of the year. I was very fortunate to be able to do so, and I found myself in the position of wanting to give back. I’d had to replace DLRS within the org for the consulting company I was working for several months previously, and the idea of creating a more performant rollup engine was an alluring way to do so. The simple RollupCalculator class I’d made seemed just a few lines away from a much simpler implementation than DLRS. Of course, iteration for apex-rollup as a whole has been ongoing since then — but it’s been a wonderful year, and I thought I’d share some thoughts & observations from having worked on a big project like this, in the hopes that some of the learnings may prove helpful to others interested in contributing to Free & Open Source Software (FOSS).

As well, I’d like to share some of the best (programming & non-programming) articles/books/podcasts I’ve consumed in the past year; they influenced me when it came time to write things here, and they certainly influenced my philosophy when it came to working on features and bugfixes for apex-rollup. Since publishing this initially, I’ve also open sourced the round robin assigner I first talked about here, and I did a little write-up on open sourcing that as well.

Some Light Reading

• Write Code Top Down — in addition to being a great article, I also love the typography on this site!
• The Broken Window Theory — I knew one thing from the get-go: I was looking for this to be a Long-Term Support (LTS) product. I never wanted to leave any part of the codebase festering.
• Good Efforts Preserve Bad Systems — goes somewhat hand-in-hand with the prior article, and I read them both around the same time. Both are good food for thought when it comes to building software; particularly software that we’d like to maintain (especially when comparing this project to other, older projects!)
• Understanding Rust Futures By Going Way Too Deep — I ended up paying homage to this article during Creating A Round Robin Assignment; in general I’d like to think that when somebody’s coming to read the Joys Of Apex, they know they’re in for a good time, and Amos’ writing style and dedication to his craft has been an inspiration for several years now. (Whether you love or hate Rust is immaterial, to me — it’s hard not to be entertained when reading his articles!)
• Blink — there was a quote in this book, “Emotion can also start on the face”, which was specifically in reference to how our bodies can influence the way that we think and feel, which I found to be fascinating when thinking about how important being in the right state of mind can be for solving difficult problems. I wrote in the Notes app on my phone after having read that sentence: “sometimes it’s when our fingers do the typing that the darndest thing happens — the answer comes out, without thinking. Recently, I was working on an issue that was covered on every code path but one — the too many SOQL rows retrieved in one transaction error. To my surprise, I found that the answer suddenly presented itself without me consciously having thought about it. The solution was surprisingly simple, and I’m not positive that I could have arrived at it in the same amount of time otherwise.”
• You 2.0 - The Value of Deep Work In An Age Of Distraction — I cannot recommend this particular episode of Hidden Brain enough. If you work on Salesforce declaritively or programmatically, there’s something in this podcast for you. This was recommended to me by a colleague at work, and I’m extremely glad I listened to it — I haven’t stopped thinking about it, since! Something I was particularly struck by, when listening, was how aligned I felt with the work done on apex-rollup and the concept of “deep work”: there were many mornings where, while examining a problem (which frequently involved dozens of tests runs, then poring through logs), that I wouldn’t write a single line of code; I would just be consumed with examining the problem from all angles. At times, this could be frustrating, but it was also consistently a source of wonder, and satisfaction.
• The New American Dream: Living Well In Small Homes — thinking about physical architecture, particularly well-designed living spaces, can be very inspirational when it comes to programming, especially on the Salesforce platform; think of the Limits class as your square footage, and the limits that we work within become exciting guide rails for building something beautiful.

The Approach

From the very beginning, I was very much settled on taking the Google Analytics approach to rollups — rather than trying to automate the process of installing rollups within an org, I’d use one line of code (or the attempted equivalent in Flow, using Apex Actions) to copy + paste, and let that serve as the jumping-off point for people to gain access to the framework as a whole. I’d seen one too many orgs brought to its knees from a combination of trying to declaratively set up DLRS (which involves DLRS creating and deploying additional triggers per object behind the scenes; a big anti-pattern!) and the tuning necessary to get the code version invoked properly without getting record-locking errors.

Looking at the initial commit, there’s a good deal of what would become the larger apex-rollup framework already scaffolded out (and you can refer to the Rollup Architecture wiki post and picture to see this). In particular, I’d like to highlight the aspirations I had from the very beginning that I view as pivotal in terms of the architecture I anticipated using:

• rollup calculations would, by default, be done asynchronously; with that being said, there would still be space within the synchronous part of the operation where some setup would be necessary
• people would be able to build their own methods for filtering the records passed in to any rollup operation using the provided Rollup.Evaluator interface. This was also a precursor to another feature that I have been very happy to work on both for apex-rollup and Nebula Logger — plugins. I was hoping to be able to make a very extensible product that could be highly customized by both admins and developers, and creating a way to easily add plugins to apex-rollup would prove to be highly rewarding!
• the concept of each kind of rollup operation being encapsulated existed already

Things that I realized almost immediately as I started working on adding other rollup operations beyond basic sums into the mix:

• the scaffolding I’d built previously didn’t support performing multiple rollup operations per Object; I’d need to have some kind of “manager” to take care of holding the rollup operations
• in addition to having strongly-typed Apex methods available, I also wanted to support CMDT-based approaches for defining Rollup operations
• I wanted Flow to be a first-class citizen within this package; indeed, many of the features that came to be developed were created for Flow first, and backported to Apex
• I was hoping to make things as Object Oriented as possible. I wanted this repository to serve as inspiration for developers in the years to come, as far as what was possible with vanilla platform features

Mistakes that I made along the way (some of which ended up being quite costly, at least in terms of time):

• I initially had created RollupControl__mdt with a lookup to Rollup__mdt, instead of the other way around. Jonathan helped point this out to me very early on in the process, but it still ended up being a breaking change for people who had already begun using apex-rollup in production. I knew I wanted to minimize the number of breaking changes it would be necesssary for people to go through; I’ve only had one other breaking change since then, and I’m hoping there won’t be any breaking changes in the future
• I spent quite a bit of time early on adding debug statements into the code, running unit tests, testing manually, looking at those debug logs, etc … 5 months in, I finally created RollupLogger and started to formally proof out which hot paths in the codebase would most benefit from having dedicated logging statements added. While logging has continued to be an area of improvement, the quality of the logs, and having consistent access to log data when troubleshooting issues for people, has proven to be immensely beneficial. TL;DR — if you’re looking to create a framework, add logging to it. Don’t forget to feature flag logging! CPU-wise, it’s an unfortunately expensive operation, and being able to turn it off is a really nice feature to have. I talk a bit more about this in Uplevel Your Engineering Skills but having logging in place is the prerequisite for “shifting left” on quality by allowing you to get fast feedback on the state of your application
• I ended up adding in the original Full Recalculation App to aid in people getting their rollups properly configured. It’s one of the only times in memory that I’ve purposefully created a non-bulkified code path — the initial implementation didn’t even offer the ability to use the Rollup Metadata CMDT records to kick off bulk recalulations. This ended up becoming an actual issue once I introduced the REFRESH context for Flow users; I thought I could be clever by delegating recalculations to the existing full recalculation code path I’d created, only to realize that with that code path not bulkified, it wasn’t possible for some of my users (notably, one with thousands of children records per parent record) to actually perform the full recalculations effectively. I don’t like to deal in absolutes — for instance, I don’t think it’s helpful to say always do this or that, especially because there was a huge amount of work that went into modifying the existing framework to be able to support full recalculations being processed in bulk. It’s not always possible to be able to anticipate things like this, and I’m not positive that I would have been able to do incrementally deliver bulk full recalculations within the same kind of timeframe in which I added in the original recalculation feature. In other words — I started off with delivering the transportation equivalent of a skateboard, and eventually was able to upgrade that deliverable to a car, but I don’t think anybody would have particularly enjoyed me delivering the car’s chassis in and of itself.
• How the lack of namespacing would contribute to the potential for clashes with other libraries. I remember the cold sweat I felt when somebody reported that both lwc-utils and apex-rollup had a LWC helper file called utils, and that my version of that file had wiped out their version of that file, rendering all their installed SOQL datatables inoperable. Yikes!

Looking back at the past 12 months, apex-rollup has provided me with a tremendous amount of satisfaction — and, occasionally, moments of high stress. I started loosely tracking the hours I was putting into my open-source work in August, and since then have racked up 200+ hours between apex-rollup and Nebula Logger. I would cautiously estimate the amount of time I’ve put into this project at 700 hours, and it’s here that I need to extend a huge thanks to my wife, who has been endlessly supportive of me putting in 2-3 hours before work, and 2-3 hours after work — especially for the first six months as I worked through the most obvious issues and feature gaps present in the project.

While I have enjoyed helping people — especially some of the awesome nonprofits I’ve been introduced to who’ve made incredible use of apex-rollup! — I was always cognizant of the fact that spending that much time on a passion project wasn’t sustainable, and I’ve been relieved and appreciative to be able to support the project at a much less harried pace at this point. Having more time has also allowed me to write here more frequently, and I was very much beginning to miss publishing the Joys Of Apex!

Some Realizations

I haven’t spent this much time on a greenfield project in several years, and it’s served as a great point of reference when thinking about code. Particularly, I’ve found a few interesting things, architecturally, and some of these realizations hit me as I re-entered the world of rock climbing this year. Climbing — and training for climbing — continues to provide valuable lessons across all walks of life.

Something I saw, time and time again while debugging code, was that sometimes the only way to figure things out was to start after the beginning. This ties into having — or working towards — good architecture, and this isn’t something specific to writing code or climbing at all. In climbing, this concept manifests as practicing one move, or one sequence, that you can’t do — until, by virtue of having tried the same thing many times, you acquire the strength and muscle memory to succeed. In programming, this frequently meant debugging only a certain section of code.

Returning to the topic of architecture — good architecture shows itself when you can assert for the presence of objects in a certain state, or print them out. If I personally couldn’t do that, sometimes that in and of itself was the problem — and in improving the code to the point where such a thing was possible, oftentimes the original problem was solved incidentally.

This idea — in climbing, practicing the individual movements; in coding, of working to isolate the problematic areas — is not new. It’s nothing that hasn’t already been said, but repetition is at the heart of learning, and so I thought to mention it here.

To put that another way, I found again and again while working on issues the difference between good and bad architecture. When the plumbing was good, I was able to move things around and easily refactor — even when this involved hundreds of changes across dozens of files. When the plumbing was bad, I was stuck working in individual methods, failing to isolate side-effects and logical permutations; unable to easily make changes for fear of breaking other parts of the system.

As an example, choosing to support things like SOQL where clauses was frequently an area that caused issues — correctly parsing things like nested conditionals (Id IN ('someId', 'someOtherId') AND (Industry = 'industry' OR AnnualRevenue > 1500000) is a relatively simple example of something that ends up being challenging to correctly handle). Is this area of the code well-encapsulated? Yes — it certainly is. Is it easy to work with or read? Absolutely not. 💯 hard no. Going back to The Tao Of Apex:

Good code works. Great code tells a story about how it works

I wrote that while working on apex-rollup, and at some level I’m sure it was a lament on some level about how difficult it can be to truly do a good job of telling the story, at a code level, for how a SOQL where clause is represented:

// in RollupEvaluator
private List<String> getSoqlWhereClauses(String localWhereClause, SObjectType calcItemSObjectType) {
  List<String> localSplitWheres = localWhereClause.split('( and | AND | OR | or )');
  this.reconstructImproperlySplitClauses(localSplitWheres, localWhereClause);
  this.recurseForNestedConditionals(localSplitWheres, calcItemSObjectType, localWhereClause);
  return localSplitWheres;
}

private void reconstructImproperlySplitClauses(List<String> localSplitWheres, String whereClause) {
  // etc ...
}

private void recurseForNestedConditionals(List<String> localSplitWheres, SObjectType calcItemSObjectType, String whereClause) {
  // ends up calling this.getSoqlWhereClauses, as promised
  // to pick apart nested conditionals -
  // would NOT recommend to a friend
  // especially when it comes to debugging!
}

Sometimes, bad architecture is forced upon us:

// in RollupAsyncProcessor
String countQuery = RollupQueryBuilder.Current.getQuery(
  rollup.calcItemType,
  new List<String>{ 'Count()' },
  'Id',
  '!=',
  whereClause
);
// a dark hack, possibly even evil -
// but there are tests that rely on only
// a single record being batched at a time
// and we can't both consume the one query
// allowed AND do the countQuery call here.
// This is because actually deferring rollups
// isn't allowed in tests - we can't rebatch or requeue
Integer outstandingItemCount = Test.isRunningTest() ? 0 : Database.countQuery(countQuery);

I wasn’t proud of that comment, but I didn’t think there was an easier way to indicate that my hands were tied by a platform limitation.

Another realization — to keep your passion project alive, it’s important to work on the areas that you really care about, even if that occasionally cuts into bugfix and feature support time. As an example, the plugin infrastructure within apex-rollup is an area of the codebase I’m particularly happy with, but not all of the plugins have been adopted by users. Last I checked, the Rollup Callback plugin (in and of itself, a feature request from an old colleague) hasn’t been installed by anybody. To some extent, this was an issue of timing; I gave the former colleague who requested the feature a barebones version of what would become this plugin since it was needed to deliver a project on time and I didn’t have the bandwidth to produce the plugin prior to their deadline. Regardless, it was an enormous amount of fun to develop the infrastructure behind this plugin in particular, and I have no regrets about the day I spent working on it.

By The Numbers

A few fun statistics for apex-rollup:

• 190/195 todos checked off from the Notes app on my phone (I considered doing a better job of tracking random thoughts related to the repo, but held off). Longest outstanding note? “Make wiki article for setting up hierarchy rollup” — though perhaps partial credit can be applied (😅) since that note originally asked for wiki articles for grandparent rollups too too!
• 106/107 issues submitted on Github closed
• 84 releases of the 2GP package since beginning packaging in April (that’s 10+ releases a month!)
• 263 unique installs of apex-rollup as a package, with a quarter of those being production-level orgs! There are still some orgs using the unpackaged metadata, as well, but I would imagine that’s a small fraction of the overall users at this point.
• 5 external contributors, of which I have to specifically call out two PRs that Jonathan put up, because they were both enormous features and I owe him a serious debt of gratitude for contributing multi-currency support and reparenting support! Stephen Reitz deserves a shoutout as well for the work he did on the SOQL Date Literal support.
• 4 plugins developed, of which the Nebula Logger plugin is by far and away the most popular!
• Incredible testing done by the community at large. In particular I’d like to extend a heartfelt thank you to a few testers in particular: Karl Berlin from the consulting world, and Justin Carter & Katherine West from the nonprofit side of things! Their feedback and willingness to pitch in to help make apex-rollup a better offering has benefitted hundreds of people worldwide
• 1 troll “gift” sent to me from Dan Donin of Cash App. I would call it artwork, but I think that would denigrate both the word “art” and the word “work”; it loosely depicts DLRS crushing Apex Rollup. I’ve already informed Dan that he’ll need to commission a better piece for next year 😂

On the subject of Nebula Logger — it has been one HECK of a year for Nebula. Jonathan and I went from tangential colleagues who hadn’t seen each other in five years to teammates again; prior to starting work on apex-rollup, I’d begun chipping away at some of the smaller Nebula issues to re-acquiant myself with the codebase after years apart from it, and it’s provided quite a few rewarding open source moments of its own over the past year! I asked Jonathan to provide some numbers and highlights — take a look at these stats!

• 152 stars on GitHub! (Most starred logging repository for Salesforce on GitHub ⭐)
• 25 releases in 2021, available as both an unlocked package and a managed package
• 74 issues closed in 2021. This includes mostly new features, although several bugfixes and optimizations have been made along the way.
• 770+ automated builds via GitHub Actions. Originally, Nebula Logger used Travis CI for the pipeline, but we switched to GitHub Actions about 9 months ago. Since then, we have been constantly expanding the pipeline’s capabilities, making testing & releases faster.
• 1,000+ unique installs, with over 300 being in production-level orgs! This includes the combined installations of the unlocked and managed packages. Much like apex-rollup, there are also orgs using the unpackaged metadata, but presumably, most orgs are now using one of the official packages.

Winding Down After A Year In Open Source

I really hope the reading/listening section will prove useful to people. Thinking about programming — and writing, in general — provides me with the motivation to do things like practice mindfulness and meditation. The benefits of pursuing opportunities for deep work have kept my impulse to program strong and healthy. Again, there are many pathways each of us can take, and I don’t like to deal in absolutes, but the least I can do is share material I found interesting in the hopes that it will prove inspiring for others as well.

2021 was an incredible year in the world of open source. To close, I’ll share a story that occurred while I was writing Replacing DLRS With Custom Rollup: while I published that article on 31 Dec 2020, the first few days of January were completely dedicated to apex-rollup; I pushed commits to main 14 (!) times in the first four days of January. I’d been so distracted with writing that the postal service taped a warning to my front door encouraging us to shovel our walkways of snow or face a fine from the city! I also found an email in my inbox from several days before — it was from my now-manager at Salesforce. They’d found my work on GitHub, and the Joys of Apex, and were looking to start a team up with strong knowledge of paired programming and TDD.

Without knowing it, I’d come home — I’d been looking to go back to work in-house for a company, and had long dreamed of working for Salesforce. Now, opportunity had come knocking. All of that is to say that if there’s one story (that I can tell, at least) that best exemplifies the huge difference that contributing to open source can make, it would be that one. A close second would be starting my first day of work and hearing we’re looking to integrate a custom logging solution into some of our applications — are there any that you can recommend? 😁 I’m doubly grateful that this year has seen Jonathan and myself able to continue what has been an incredibly fruitful and rewarding collaboration!

Lastly, a shout out to my colleague Sébastien Colladon, who undertook the massive project of reading every post I’d written and provided me with invaluable feedback on many of the articles. His Async Apex Linkable repo is one that I’ve been meaning to write about, and to dive into myself. If you enjoyed this article, and want to read more on the subject of engineering in general, I would recommend Uplevel Your Engineering Skills as well as The Life & Death Of Software.

Here’s to hoping that for all of you reading out there that 2021 ends well; that in the midst of a global pandemic there are notes of optimism amidst the highs and lows of our lives in COVID; that 2022 is filled with new opportunities and rewarding moments of learning. Till next time — James.

For more on open source, be sure to keep an eye on the Salesforce Developer Podcast, where I'll be sharing my open source journey soon, as well as in Building An Apex Portfolio. I’ve also added:

• a second reflection piece from 2022: Two Years In Open Source
• a third reflection piece from 2023: Three Years In Open Source