The blurred line between overengineering and anticipating change

From the rise of personal computers up to the last decade, our hardware was so limited that engineers’ main challenge was letting everything fit in and squeezing every single transistor of those chips that now we perceive as prehistoric. To this day I’m still mind-boggled by the optimizations video games came up with between the late 90’s and early 2000’s: to name a few, Quake’s fast inverse square root and Crash Bandicoot literally hacking the PS1 to swap data faster.

However, times change, and software follows: with engineers no longer needing to leverage low-level tricks to achieve the impossible, the focus shifted towards robust, maintainable, predictive, and automatically-tested software.

A shallow dive into internal software quality

When talking about good code, we typically mean code that’s easy to pick up, understand, and change, and that follows the software engineering principles, that include:

• Abstraction: hiding complexity behind a simple interface.
• Modularity: organizing into independent components.
• Separation of concerns: dividing into areas of functionality.
• Anticipation of change: designing to accommodate future modifications.
  This is my favorite aspect of software design and the main focus of this post.

When you’re kicking off a new project, it’s tempting to just start coding and get things done quickly. Following these principles can feel like an unnecessary burden, but that’s where things usually backfire: after the initial excitement of dozens of new features, the codebase becomes a tangled mess, and each new feature takes more and more time to implement, as you navigate through your delicious spaghetti code.

I learned about the following chart in my uni classes, and I wish I had seen it sooner:

xychart-beta
	x-axis "Time (days)"
	y-axis "Number of features shipped"
	line [0.0, 6.931471824645996, 10.986123085021973, 13.862943649291992, 16.094379425048828, 17.91759490966797, 19.4591007232666, 20.794414520263672, 21.972246170043945, 23.02585220336914, 23.978954315185547, 24.84906768798828, 25.649492263793945, 26.390573501586914, 27.080501556396484, 27.725887298583984, 28.332134246826172, 28.903717041015625, 29.44438934326172, 29.95732307434082, 30.44522476196289, 30.910425186157227, 31.354942321777344, 31.78053855895996, 32.188758850097656, 32.580963134765625, 32.958370208740234, 33.322044372558594, 33.67295837402344, 34.0119743347168, 34.33987045288086, 34.6573600769043, 34.9650764465332, 35.26360321044922, 35.55348205566406, 35.83518981933594, 36.10917663574219, 36.37586212158203, 36.635616302490234, 36.8887939453125, 37.135719299316406, 37.3766975402832, 37.612003326416016, 37.841896057128906, 38.066627502441406, 38.286415100097656, 38.5014762878418, 38.71200942993164, 38.9182014465332, 39.12023162841797, 39.31825637817383, 39.51243591308594, 39.702919006347656, 39.88983917236328, 40.073333740234375, 40.253517150878906, 40.430511474609375, 40.60443115234375, 40.77537536621094, 40.94344711303711, 41.10873794555664, 41.27134323120117, 41.43134689331055, 41.588829040527344, 41.743873596191406, 41.896549224853516, 42.04692840576172, 42.19507598876953, 42.341064453125, 42.48495101928711, 42.626800537109375, 42.76666259765625, 42.90459442138672, 43.040653228759766, 43.17488098144531, 43.307334899902344, 43.438053131103516, 43.56708908081055, 43.69447708129883, 43.82026672363281, 43.94449234008789, 44.06719207763672, 44.18840408325195, 44.30816650390625, 44.426513671875, 44.54347229003906, 44.659080505371094, 44.77336883544922, 44.8863639831543, 44.99809646606445, 45.10859680175781, 45.21788787841797, 45.32599639892578, 45.43294906616211, 45.53876876831055, 45.64348220825195, 45.747108459472656, 45.84967803955078, 45.95119857788086, 46.05170440673828]
	line [0.09090909361839294, 0.7272727489471436, 2.454545497894287, 5.818181991577148, 11.363636016845703, 19.636363983154297, 31.18181800842285, 46.54545593261719, 66.2727279663086, 90.90908813476562]

Overengineering is a myth

It’s worth mentioning that, when referring to overengineering, we’re in the context of software architecture. We’re not discussing it in the sense of distributed systems and microservices, as that isn’t my field of expertise.

As an advocate for clean code, I often ask myself whether I’m overengineering things. Whenever I believe I’ve crossed that line, I’m proven wrong as soon as I need to add new functionality on top of what I built earlier.

At that point, I realize the design choices I’d made were actually anticipating change: even without foreseeing the exact change or having any gut feeling about how the software would evolve 6 months down the line, those choices still turn out to be the right ones.

A practical example

When using the Quarkdown CLI, the quarkdown create command generates a new Quarkdown project with a predefined structure.

The requirements for this command were:

• --main-file <name>: option to specify the name of the main file. Default is main.
• Multiple prompts to gather name, description, document type, and other properties.
  These properties are then used in the main file as metadata: .docname {...}, .docdescription {...}, .doctype {...}, and so on.
• Unless the --empty flag is provided, the main file should contain sample content to help the user grasp the syntax and learn how to use the CLI. Also, an images directory should be created, containing a sample image referenced in the main file.

At this point, you might think the implementation is straightforward: create the main file with the specified name, inject properties, and optionally add the sample content and the images directory:

fun createProject(
    mainFileName: String,
    empty: Boolean,
    info: DocumentInfo
) {
    val mainFile = File(mainFileName + ".qd")
    mainFile.writeText(
        """
        .docname {${info.name}}
        .docdescription {${info.description}}
        .doctype {${info.documentType}}
        
            ${if (!empty) """
                # ${info.name}

                Welcome to Quarkdown!

                ## Compiling

                ...
            """.trimIndent() else ""}
        """.trimIndent()
    )
    
    if (!empty) {
        val imagesDir = File("images")
        imagesDir.mkdir()
        // Add sample image to the images directory
    }
}

Although it’s concise and meets the requirements, this implementation is not maintainable:

• If we want to add more properties in the future (and we will), we’d need to modify writeText’s raw string.
• If we want to change the sample content, we’d need to modify yet another nested raw string.

This is what anticipation of change is all about: designing code that can accommodate future modifications without having to rewrite large portions of it.

Instead, here’s another approach:

• Load the contents from a JTE (Java Template Engine) template, a simple text file with placeholders.
• Instead of creating a file right away, return an OutputResource, Quarkdown’s abstraction of a file to be created.

fun createProject(
    mainFileName: String,
    empty: Boolean,
    info: DocumentInfo
): OutputResource {
    val templateContent = loadTemplate("main.qd.jte")
    val renderedContent = renderTemplate(templateContent, info, empty)
    return OutputResource(mainFileName + ".qd", renderedContent)
}

The final architecture

Well, that’s straightforward and clean. Why would that be overengineered? Because that’s not what I went with! I instead went for this monster of a solution:

• ProjectCreator: the orchestrator. It doesn’t know how to create templates or what initial content to include. Instead, it delegates these responsibilities to two injected strategies:
• ProjectCreatorTemplateProcessorFactory: a factory that creates JTE template processors and injects the document info. It also defines which files to generate via createFilenameMappings(), a map of file names to their own template processor.
• ProjectCreatorInitialContentSupplier: a strategy that provides the sample content and additional resources (e.g. images). The default implementation loads a separate .jte template with the sample code, plus the image.
  If --empty is provided, the Empty supplier is used, which returns empty content and no resources.

When createResources() is called, ProjectCreator iterates over the factory’s file mappings, processes each template, and collects all results.

class ProjectCreator(
    private val templateProcessorFactory: ProjectCreatorTemplateProcessorFactory,
    private val initialContentSupplier: ProjectCreatorInitialContentSupplier,
    private val mainFileName: String,
) {
    fun createResources(): Set<OutputResource> {
        val resources =
            this.templateProcessorFactory.createFilenameMappings()
                .map { (fileName, processor) ->
                    TextOutputResource(
                        fileName ?: mainFileName,
                        processor.process(),
                    )
                }

        return buildSet {
            addAll(resources)
            addAll(initialContentSupplier.createResources())
        }
    }
}

classDiagram
    direction TB
    class ProjectCreator {
        -mainFileName String
        +createResources() Set~OutputResource~
    }
    class ProjectCreatorTemplateProcessorFactory {
        <<interface>>
        +createFilenameMappings() Map~String?, TemplateProcessor~
    }
    class DefaultProjectCreatorTemplateProcessorFactory {
        -info DocumentInfo
    }
    class ProjectCreatorInitialContentSupplier {
        <<interface>>
        +templateContent String
        +createResources() Set~OutputResource~
    }
    class EmptyProjectCreatorInitialContentSupplier {
        +templateCodeContent = null
        +createResources() = emptySet
    }
    class DefaultProjectCreatorInitialContentSupplier {
        +templateCodeContent
        +createResources() images
    }
    ProjectCreator --> ProjectCreatorTemplateProcessorFactory
    ProjectCreator --> ProjectCreatorInitialContentSupplier
    ProjectCreatorTemplateProcessorFactory <|.. DefaultProjectCreatorTemplateProcessorFactory
    ProjectCreatorInitialContentSupplier <|.. DefaultProjectCreatorInitialContentSupplier
    ProjectCreatorInitialContentSupplier <|.. EmptyProjectCreatorInitialContentSupplier

What proved me wrong

I knew this solution was slightly excessive, but I went with it anyway. Overengineered, until, about one year later, I added support for a new document type: docs. This new type allows Quarkdown to generate multi-page technical documentation and wikis, which have a different structure and different sample content compared to the other types. This is the structure of a docs project that should be generated:

On top of that, page-N.qd files shouldn’t be generated if the --empty flag is provided.

If I had stuck with the naive implementation, that would have been a nightmare to keep up with: just imagine the amount of conditional logic and nesting to accommodate the new files to be generated.

Instead, with my implementation it was just a matter of adding two small classes:

• DocsProjectCreatorTemplateProcessorFactory: overrides createFilenameMappings() to generate two files instead of one: _setup.qd and main.qd.
• DocsProjectCreatorInitialContentSupplier: reuses the default’s sample code content and provides the additional page resources (_nav.qd, page-1.qd, page-2.qd, page-3.qd).

No changes to ProjectCreator itself were needed: it remained completely unaware of docs projects. The CLI command simply picks the right strategy based on the document type, and the rest falls into place:

return ProjectCreator(
            templateProcessorFactory =
                when {
                    isDocs -> DocsProjectCreatorTemplateProcessorFactory(documentInfo)
                    else -> DefaultProjectCreatorTemplateProcessorFactory(documentInfo)
                },
            initialContentSupplier =
                when {
                    noInitialContent -> EmptyProjectCreatorInitialContentSupplier()
                    isDocs -> DocsProjectCreatorInitialContentSupplier()
                    else -> DefaultProjectCreatorInitialContentSupplier()
                },
            mainFileName,
        )

classDiagram
    direction LR
    ProjectCreator --> ProjectCreatorTemplateProcessorFactory
    ProjectCreator --> ProjectCreatorInitialContentSupplier
    ProjectCreatorTemplateProcessorFactory <|.. DefaultProjectCreatorTemplateProcessorFactory
    ProjectCreatorTemplateProcessorFactory <|.. DocsProjectCreatorTemplateProcessorFactory
    ProjectCreatorInitialContentSupplier <|.. DefaultProjectCreatorInitialContentSupplier
    ProjectCreatorInitialContentSupplier <|.. EmptyProjectCreatorInitialContentSupplier
    ProjectCreatorInitialContentSupplier <|.. DocsProjectCreatorInitialContentSupplier

Takeaway

This example shows why I don’t believe overengineering is a thing in software architecture. Strategy pattern, factory, supplier: they all seemed like overkill for a simple project generator. But when the requirements evolved, the architecture held up without changes to the core ProjectCreator.

This was a relatively simple and small example, but the same principle easily scales up to larger and more complex systems.