Best Practices are (at least partially) Scale Dependent

Something that seems to hold true across all organisation sizes is that (at least a sizeable subset of) software developers are opinionated (with perhaps the number of opinionated developers increasing with scale). Opinions exert a considerable influence on software development practices, with many developers interested in “what are the best practices”. After many decades as a profession, it seems fair to say that there is now a reasonable base set of best practices that all software development teams should adopt. It’s less clear if everyone would agree about this set of best practices (perhaps everyone can agree that version control software is a good idea?) and it’s fairly clear that a random individual’s set of best practices would not overlap. Some of this reflects differences in taste (individuals vary in how they weight the value of different aesthetics) and some of this reflects differences in experience (the adopted best practice has been around insufficiently long for the profession to say whether it is or is not a best practice). These two factors, however, leave quite a bit of the variation unexplained.

A Toy Model

A simple model (displayed below, produced using excalidraw) of best practices is that a best practice has an adoption cost and long-run returns. The adoption cost and the long-run returns, however, are often highly dependent on an organisation’s scale. More specifically, adoption costs are of varying sizes and must be paid upfront. Long-run returns are more variable and include both maintenance costs and the benefits paid to each individual. Maintenance costs include both the effort all individuals need to invest into maintaining the best practice but also the cost of maintaining any associated tooling.

At a small scale, the upfront costs of adopting a given practice are often the bottleneck to adopting that practice. If a given practice requires the dedicated attention of one of two engineers for two weeks, it’s far less likely to be adopted. For a best practice to be scale independent, it needs to have effectively no upfront cost.

After considering upfront costs, the other question is “what are the returns to adopting this practice”.¹ Importantly, many practices benefit from network effects. For some, a practice only becomes useful at a given scale, while for others the benefits grow with scale.

Examples

Version Control Software

Why is version control software (VCS) adoption effectively ubiquitous? The upfront costs of adopting VCS have been incredibly low due to services such as GitHub and Bitbucket. The effort necessary to learn to use VCS is generally paid by the individual ², making it *effectively ree for the organisation to adopt. The benefits to the organisation are large and immediate (protection against potentially catastrophic loss of work).

The above, of course, understates why organisation’s use VCS. VCS is not used just to protect against loss of work, it’s used to manage versions of code, centralize where code is stored, and allow multiple developers to collaborate on a single codebase. VCS enables developers to more effectively review code (via features such as pull requests), to search code (via features such as GitHub’s code search), or automatically flag problematic dependencies in code (such as via GitHub’s dependabot). These features will be used differentially across organisations, with more advanced uses emerging as the organisation scales.

Importantly, less basic GitHub features likely are not universally adopted. It seems likely that effectively all organisations should use code review ³, but it’s not clear that effectively all do, potentially because of upfront costs. Universities seem to succeed in teaching developers how to be basic users of VCS, but “less basic” use cases do not seem to be universally taught ⁴. As one moves beyond code reviews, the benefits of more advanced features likely saturate for small organisations, but certain features become more valuable as an organisation scales.

Issue Tracking

Issue tracking software is likely considered another best practice which is widely adopted across the industry. Once again, the upfront costs are small (many services exist) and tools have fast learning curves ⁵. As organisations scale, the maintenance costs of issue trackers grow. More users means that users are likely to either submit duplicate tickets or need to spend more team looking to see if a ticket has been submitted before submitting an issue. This requires changes in how issue trackers are used.

Unit Testing

Unit testing advocates will argue that unit testing is a best practice which should be widely adopted. Instead, it seems likely that the benefits of unit testing are scale dependent. In small organisations, unit tests are less valuable than in large organisations. A simple model is that unit tests are valuable when they prevent a sufficient number of regressions relative to the number of false positives and their associated maintenance and development costs. Since (on average) fewer developers will be modifying a given codebase, the average developer will be more experienced with the codebase, reducing regression frequency (to whatever degree).

Some more specific unit testing practices (aiming for 100% coverage) may make sense in certain contexts, but are likely a poor investment for many organisations.

Custom Tooling

Human manipulation of application configuration or data is incredibly inefficient but, sometimes, is necessary. The development of organisation-specific tooling can reduce the costs of human manipulation, but often require significant upfront investments. This means that small organisations generally rely on practices which scale poorly but are locally optimal. As they grow, they need to modify their culture such that they begin to invest in organisation-specific tooling. At a certain scale (think Google/Microsoft/AWS), organisations benefit from developing organisation-specific issue trackers, code review tools, etc.

Of note, there is a useful middle ground between full-featured custom tooling and no tooling. For REST APIs, web frameworks which feature automatic generation of swagger UIs (such as FastAPI) from API paths are incredibly useful, even for small organisations.