Resiliency against change & avoiding regression

The real cost of software isn’t the initial development, but the maintenance over time. In time, the requirements will change, there will be new feature requests and the business might change direction. With all those undeniable changes coming to the code, there’s a real need to make the systems more resilient to inevitable change.

From Building Resilient Frontend Architecture by Monica Lent

Software solution resiliency refers to the ability of a solution to absorb the impact of a problem in one or more parts of a system while continuing to provide an acceptable service level to the business. While in an ideal world, a resilient system would be able to deal with any problem in a way that would have no negative impact, but even with proper design and testing, it’s likely that some user population and requests will be impacted by a failure.

By IBM

As software developers, we are building tests as safeguards for the automation that we’re either trying to build and/or maintain. Those tests are proof that your code works as you expect it to work.

One thing I feel that doesn’t get enough love is the testing component of a software. The thing with tests is that creating good tests isn’t the easiest thing to do. But it’s a ‘necessary evil’.

Without them, it would be many times harder to do the kind of work that we’re doing today. When a bug suddenly appears or rather, is made apparent through a change, creating regression tests is a solid approach to make sure that they don’t reappear again. Also, while doing a refactoring of a feature, they’re also proof that your code works just as it used to.

Testing small chunks with unit testing

I am guessing that most of my readers are at least familiar with unit testing here. But for those who aren’t, this quote should help you understand what unit testing is.

Essentially, a unit test is a method that instantiates a small portion of our application and verifies its behaviour independently from other parts. A typical unit test contains 3 phases: First, it initializes a small piece of an application it wants to test (also known as the system under test, or SUT), then it applies some stimulus to the system under test (usually by calling a method on it), and finally, it observes the resulting behaviour. If the observed behaviour is consistent with the expectations, the unit test passes, otherwise, it fails, indicating that there is a problem somewhere in the system under test. These three unit test phases are also known as Arrange, Act and Assert, or simply AAA.

By Toptal

From the above quote, you can see that a unit test is here to test only small manageable chunks of your software. Don’t try to test too many at once, it’ll make the test easier to maintain over time.

Unit testing is an essential instrument in the toolbox of any serious software developer. […] Unit tests are quite easy; the real problems that complicate unit testing, and introduce expensive complexity, are a result of poorly-designed, untestable code. […] Writing unit tests and generating testable code is not just about making testing less troublesome, but about making the code itself more robust, and easier to maintain.

By Toptal

Property-testing to test your specs

The issue with unit testing is that you’re mostly trying to capture a single scenario with a specific input being tested. This means that you either don’t completely get to test your application inside and out or, you’re going to write a lot of tests in order to get bigger data coverage. In order to solve this, we have access to property testing.

Property-based tests are designed to test the aspects of a property that should always be true. They allow for a range of inputs to be programmed and tested within a single test, rather than having to write a different test for every value that you want to test. Property-based testing was popularized by the Haskell library QuickCheck. Tests that are similar to property-based tests include fuzz testing or the use of example tables in ATDD/BDD-style testing.

From Tech Beacon

The answer is to create tests that focus on the properties of the function – the “requirements”. These properties should be things that are true for any correct implementation.

From F# for fun and profit

So, in a few words, you can write way fewer tests than usual and have a library evaluating your codebase with a range of inputs that you do not have to provide. This will lead to a better testing infrastructure. Sometimes, it is hard to think of every possible edge case for your code.

Capturing your specs through your types

A little something extra that isn’t in the scope of this article. We’ve been talking about testing a lot but in the last section, we’ve been talking about testing your software requirements.

One thing that can be extremely hard to document and translate to code are the business requirements. I believe that being able to embed your requirements directly into your data types makes it easier to reason with them. It will lead to less external documentation outside of the codebase. Your code becomes the documentation that new hires are required to read & thoroughly understand.

Creating a test fixture with randomized input

For the uninitiated, here’s a definition of a test fixture from Wikipedia.

A test fixture is an environment used to consistently test some item, device, or piece of software. Test fixtures can be found when testing electronics, software and physical devices.

Wikipedia

Where things become interesting is when you move from setting up values by hand to values that are set automatically. When we set values by hand, we’re trying to capture a specific scenario but that’s precisely what we’re trying to avoid. A randomized data environment testing your code is a good way to find the potential flaws in implementation.

By building a test fixture, one that is immutable to change (hello functional programming), you now have access to a testing environment that is reusable for any number of tests. There’s a lot of power into this sort of approach. It takes time and effort to do so but once it’s established, then you can truly build uniform tests that are dependant on this testing setup.

There’s a popular library in .NET that you can use called FsCheck.

FsCheck is a tool for testing .NET programs automatically. The programmer provides a specification of the program, in the form of properties which functions, methods or objects should satisfy, and FsCheck then tests that the properties hold in a large number of randomly generated cases. While writing the properties, you are actually writing a testable specification of your program. Specifications are expressed in F#, C# or VB

FsCheck read.me

We’ve now covered the fundamental concepts for this article and we can now proceed to apply those into code. For this, I will be using functional programming and F# to showcase what I’ve been discussing.

Ex: RPG-style game

To accompany this post, I spent some time building up a codebase for a proof of concept (PoC). Let’s start with what you can expect to find in the codebase.

This PoC is focused on tactical role-playing games. You can expect to find data types for units, weapons, armours, spellbooks, an inventory for the party and a game store.

You can find the repository here.

Purchasing from the store

As the player, you want the units you’re controlling to have the best merchandise that’s possible at the current point you find yourself in the game.

Testing every outcome and possibilities of this scenario can be quickly complicated to set up. First, you need a fake party that can make purchases from a merchant store. Let’s first define what is a tactical game unit and then create a fake party within the testing fixture. You can read the code in detail from the link shared earlier.

type GameUnit = {
    Name: string
    Lvl: int16
    Job: CharacterJob
    CurrentState: CharacterState
    TotalXp: int 
    SkillPoints: int16
    Equipment: Equipment list
    Consummables: ConsumableItem list
}

type GameTestFixture = {
    Party: GameUnit list
    Store: GameStore
    PartyInventory: Inventory
}

let createSingleValueWithGenerator<'Value> (generator: Gen<'Value'>) =
    generator |> Gen.sample 1 1 |> List.head

let unitPartyFixture =
    let generateGameUnit() =
        let equipmentListValues = [0 .. random.Next(1, EquipmentSlotLimit)]
        let consummableItemListValues = [0 .. random.Next(1, ConsummableItemSlotLimit)]
        {
            Name = createSingleValueWithGenerator<string> stringGenerator
            Lvl = createSingleValueWithGenerator<int16> int16Generator
            CurrentState = createSingleValueWithGenerator<CharacterState> characterStateGenerator
            TotalXp = random.Next(0, MaxExperiencePointBeforeLevelingUp)
            SkillPoints = random.Next(0, MaxSkillPoints)
            Job = createSingleValueWithGenerator<CharacterJob> characterJobGenerator
            Equipment = List.map(fun _ -> createSingleValueWithGenerator<Equipment> equipmentGenerator) equipmentListValues
            Consummables = List.map(fun _ -> createSingleValueWithGenerator<ConsumableItem> consummableItemGenerator) consummableItemListValues
        }

    let partySize = random.Next(MinPartySize, MaxPartySize)
    List.map(fun _ -> generateGameUnit()) [1..partySize]

In the snippet above, we now have a fixture for a fake party composed of 3 to 12 units. Then we’re going to need to set up a merchant store from which we can purchase consumable items (i.e. a potion), weapons (i.e. a sword) & armoury (i.e. a helmet).

Previously, we were talking about the requirements. Let’s see what can be applied here. When a customer (read party member), enters a store, they have many obligations to respect in order to buy an item such as:

• Is the team inventory full?
• Is the team inventory at its weight limit?
• Can the specific unit buy any more equipment?
• Can the specific unit buy any more consumable items?

This can lead quickly to complicated safeguards put inside the code and trying to make sense of them in the future if there isn’t any proper documentation can be a hard feat to accomplish when debugging code. Luckily, F# is a language in which it is pretty easy to embed requirements in a succinct manner.

type PurchaseFromStore = (int16 * GameItem) -> Inventory -> GameUnit -> (Inventory * GameUnit)

From the snippet above, we have created a type that holds the requirements for purchasing a specific quantity of a given game item from the store provided that we show our inventory and our specific unit.

Let’s keep in mind that it only represents a function signature and by no means it represents the implementation of a purchase which you can find below for the PoC.

let purchaseFromStore : PurchaseFromStore =
        fun itemAndQty inventory gameUnit ->
            let quantityInt = (fst itemAndQty) |> int
            let itemToBePurchased = (snd itemAndQty)

            if gameUnit.CanPurchaseEquipment |> not then
                (inventory, gameUnit)

            elif (quantityInt + gameUnit.Equipment.Length) < EquipmentSlotLimit then
                (inventory, gameUnit)

            elif (quantityInt |> float) * itemToBePurchased.Weight + inventory.CurrentWeight < inventory.MaxWeight then
                (inventory, gameUnit)

            else
                storeFunds <- storeFunds + (fst itemAndQty)

                let totalPurchaseOrder = itemToBePurchased.Price * quantityInt
                let addedWeight = itemToBePurchased.Weight * (quantityInt |> float)
                let gameUnit =
                    match itemToBePurchased with
                    | Consumable consummable ->
                        let storeItems = [1..quantityInt] |> List.map(fun _ -> consummable)

                        { gameUnit with Consummables = gameUnit.Consummables |> List.append storeItems }

                    | Equipement equipment ->
                        match equipment with 
                        | Weapon weapon ->
                            let storeItems = [1..quantityInt] |> List.map(fun _ -> Weapon weapon)

                            { gameUnit with Equipment = gameUnit.Equipment |> List.append storeItems }

                        | Armor armor ->
                            let storeItems = [1..quantityInt] |> List.map(fun _ -> Armor armor)

                            { gameUnit with Equipment = gameUnit.Equipment |> List.append storeItems }

                let inventory = {
                    inventory with 
                        Funds = inventory.Funds - (totalPurchaseOrder |> int16)
                        CurrentWeight = inventory.CurrentWeight + addedWeight
                }

                (inventory, gameUnit)

All that is left is to complete the testing fixture (found on the repository) & create both unit tests and property-tests for our game.

let gameFixture = {
    Party = unitPartyFixture
    Store = storeFixture
    PartyInventory = inventoryFixture
}

I will continue the adventures for this PoC at a later time. This will give me the chance to clean up the implementation a bit and add some tests to back it up. After that, based on feedback and my motivation, we can see how much further we’ll push this codebase.

Concluding remarks

The entire article has been focused on using randomized testing input and creating an immutable testing fixture to help build your tests quicker. Yes, the codebase is written in .NET and specifically in F#, but this applies to any kind of Turing-complete languages such as Python, C++, C# or Rust and Javascript.

I am mentioning this so we, as software developers, broaden our vision and try to learn from material outside our technical stack. There are always lessons to be learned out there somewhere. Furthermost, it’s not because some material you find online is old that it’s entirely out-dated. You may find insightful material nonetheless.

For those new to F# and/or functional programming, I invite you to take a quick glance at the last 3 references listed in the next section!

Don’t hesitate to reach out if you have comments or questions 🙂

Helpful references

• A Guide to Software Resilience Testing
• Software solution resiliency guidelines can help prevent bad behaviour
• An introduction to property-based testing
• Why use F# series
• The “Thinking functionally” series
• The “Designing with types” series

You can follow my tech journey on Twitter & GitHub and subscribe to my blog.

Until next time,

Kevin A.

• C# & F#
• F#
• Unit testing