I’m Blake. I write about probability, stochastic processes, Python tooling, and the practical side of building things with code. More about me →

  • Better config files for python ML experiments

    ML experiments essentially boil down to a single script, for example:

    # ml_experiment.py
    #
    # Executed with `python ml_experiment.py`
    #
    
    
    def main():
        dataloader = ...
        model = ...
        loss_function = ...
        optmizer = ...
        n_epochs = ...
    
        losses = []
        for epoch in range(n_epochs):
            epoch_error = 0.0
            for x, y in dataloader:
                y_hat = model(x)
                loss = criterion(y_hat, y)
                loss.backward()
                optimizer.step()
                optimizer.zero_grad()
                epoch_error += float(error)
            losses.append(epoch_error)
    
        ...
    
    
    if __name__ == "__main__":
        main()

    This works for a single experiment, but in development we always use multiple experiments. We should copy and paste all the common iteration and optimisation code across every experiments, since this is extremely error prone. Instead we should opt for extracting all the common training logic into a single script and specify the experiment-specific settings (i.e. dataloader, model, loss function, etc.) in a config file that we can load:

    # train.py
    #
    # Executed with `python train.py <CONFIG_PATH>`
    #
    import argparse
    
    
    def parse_args() -> argparse.Namespace:
        parser = argparse.ArgumentParser()
        parser.add_argument("config_path", type=str, nargs="?")
        return parser.parse_args()
    
    
    def main():
        args = parse_args()
        config_path = args.config_path
    
        dataloader = load_dataloader_from_config(config_path)
        model = load_model_from_config(config_path)
        loss_function = load_loss_function_from_config(config_path)
        ...
    
    
    if __name__ == "__main__":
        main()

    We could use a format such as JSON or YAML, but this introduces new problems:

    • How can we effectively select between different python objects (i.e. different models and different args and kwargs) without accidentally writing a horrifically ugly and error prone load_from_* functions?

    Who seriously wants to write a bunch of convoluted parsing code?! I’ve written a few of these and seen a couple of implementations out in the wild, but it’s never sat well with me — I would much rather focus on the actually important machine learning code!

    Why don’t we skip the config interpreters, and just write our configs as python scripts? A couple of upsides:

    • No convoluted parsers to write
    • Natural and interpretable parameter and object definition
    • Ability to write more bespoke object initiation for certain experiments

    But the downside is that we’d have to hardcode the imports at the top of the file:

    # train.py
    #
    # Executed with `python train.py`
    #
    
    # Question: do I have to hardcode this?
    # What if I want to select a different config?
    # Also python imports can be a bit annoying sometimes.
    from ml_experiment.config import (
        dataloader,
        model,
        loss_function,
        optimizer,
        n_epochs,
    )
    
    
    def main(): ...
    
    
    if __name__ == "__main__":
        main()

    But there’s a way around hardcoding the imports by using the built-in methods of importlib. I learned this snippet from one of my last jobs (thanks to Tim Esler and David Norrish):

    # src/utils.py
    #
    from importlib.util import module_from_spec, spec_from_file_location
    from pathlib import Path
    from types import ModuleType
    
    
    def import_script_as_module(script_path: str | Path) -> ModuleType:
        """Execute a python script and return object where symbols are defined as attributes.
    
        Args:
            config_path: Path to a python script.
        """
        config_path = Path(script_path)
        name = Path(config_path).stem
        spec = spec_from_file_location(name, config_path)
        module = module_from_spec(spec)
        spec.loader.exec_module(module)
        return module

    This is effectively a way to dynamically load a .py file as if it were a normal module, without needing it in sys.path.

    How it works:

    • spec_from_file_location(name, location)
      • Creates a ModuleSpec object that describes how to import a module from the given file path.
      • Tells Python where the module is and what its name should be.
    • module_from_spec(spec)
      • Creates a new, empty module object based on the ModuleSpec.
      • Prepares a module object that can later be populated by executing the actual code.
    • spec.loader.exec_module(module)
      • Loads and executes the code from the given file into the module object created above.
      • Actually runs the script, populating the module’s namespace with the variables, functions, classes, etc. defined in it.

    So now I can write my experiment runner as:

    # train.py
    #
    # Executed with `python train.py <CONFIG_PATH>`
    #
    import argparse
    
    from src.utils import import_script_as_module
    
    
    def parse_args() -> argparse.Namespace:
        parser = argparse.ArgumentParser()
        parser.add_argument("config_path", type=str, nargs="?")
        return parser.parse_args()
    
    
    def main():
        args = parse_args()
        config = import_script_as_module(args.config_path)
    
        dataloader = config.dataloader
        model = config.model
        loss_function = config.loss_function
        ...
    
    
    if __name__ == "__main__":
        main()

    And my config file can be written as a simple modularisable Python script:

    # config.py
    #
    
    model = MyModel(arg1, arg2, ...)
    
    transforms = [...]
    dataset = Dataset(transforms, ...)
    dataloader = DataLoader(dataset, ...)
    
    ...

    Taking this approach, we end up with an experiment setup that is much more interpretable and maintainable.

    And I don’t have to write a single line of YAML, which is always a win.

  • How I review pull requests

    Collaboration is often the bottleneck in software development. As addressed in the classic “Mythical Man-Month” by Fred Brooks, Moore’s law leads to faster hardware each decade, but in spite of that the rate of software delivery stays relatively constant. The bottleneck is never the hardware — it’s the people and communication processes.

    Git is the most widely used tool for distributed version control, and Github provides a platform for collaboration on git repositories. Our workflows generally consist of gathering requirements, studying existing codebases, and making changes without toppling the proverbial Jenga tower. We make changes on a new branch on the git repository, and then we make a pull request (PR) before merging the branch into the main branch of the project so that someone else on the team can check the change is appropriate to merge.

    For many teams, this is often where things come to a halt. Code review is an essential part of the process to ensure that contributions are agreed upon across the author and team members. It’s also a spot for reviewers to interrogate the code and check that the contributions won’t break anything.

    Today I want to share a bit about my review process to try and articulate what goes into a good review, and infer strategies for submitting better pull requests.

    The description

    Reading the title of the PR is always a great first step. When the title is written like a conventional commit, I know exactly what context I need to switch to. I also find assigning tags such as Low impact, Medium impact, and High impact great for judging at a glance how much time and effort I’ll need to set aside to review the PR.

    The description should have a short paragraph or two (or some dot-points) describing the change in more depth. In my opinion, a great PR description tells me:

    • What is changed
    • How it works
    • Why it was designed this way

    This is also a great spot to document what didn’t work. These decisions are rarely documented, yet they often contain valuable insights!

    If the description is too long, that’s a great indication that the change is too big — and will thus take much longer to review. In keeping with “Curly’s law”:

    Things should do one thing, and one thing only

    Excellent advice for writing good functions, classes, and modules. I can review PRs that align with this principle much faster.

    Testing

    When there’s evidence that the change was tested by the author, I feel more confident about the PR. But, if the author additionally provides me with instructions on how to test the PR myself, I feel even more confident! Explaining how to test a change forces the author to think critically, often surfacing issues before the reviewer even sees the code.

    Unit tests also give me more confidence in the change, especially if they’re well written. This is a topic for another post, but “Test Driven Development: By Example” by Kent Beck has some great advice on this.

    Reading the code

    I’ve studied the PR description and I’ve ensured that it’s well tested. I’ve also checked out the branch on my machine and checked that it works. If no red flags have emerged, I move on to reading the code.

    Ideally, I’d like to review the change in Github. If the change is small enough, the diffs in the PR don’t put too much cognitive load on me and thus I can review it faster. That being said, tiny changes can often result in higher failure rates. I think both authors and reviewers can get over-confident in changes that only span a few lines, and thus they don’t get scrutinised as much. This is a bias we all have, and we just need to be a bit more vigilant about that!

    Most of the time, I’ll open a text file to jot down my thoughts during a review. In Vim I’ll run:

    :read!git diff main --name-only
    :norm! I- [ ]

    Which gives a nice little checklist in my file:

    - [ ] README.md
    - [ ] src/floob.py
    - [ ] src/paths.py
    - [ ] test/test_floob.py

    Then I’ll open each file one by one, jump around definitions and references. Each comment and question that comes to mind, I’ll keep track of them in my running document:

    src/floob.py
    Optional style comment, early returns are great
    ~~~python
    if condition:
        LOGGER.info("Floobed is already wumboed")
        return
    
    LOGGER.info("No wumbo found")
    df = foo(floob, bar=2)
    df["wumbo"] = ...
    ~~~

    This gives me a good way to edit all my comments before posting the more important ones on the PR review on Github.

    Lessons

    What can we do proactively to ensure that we write small PRs so that we can get changes reviewed faster? I used to start a branch, solve the problem, and submit a PR. But this approach is often haphazard, because it’s difficult to be vigilant about the scope and keep all the commits and diffs on-topic. As tempting as it is to bundle some refactors in to my PRs, this almost always adds friction to the review process.

    Nowadays I try spending more time planning before diving into the code:

    • Lots of studying of the existing codebase
    • Think about the problem, write down some strategies
      • Diagrams for each approach, visualise the control flow
      • Weight up the pros and cons of each approach
    • Do some mockups, write some scaffolding code to illustrate how your change will fit in (and what other changes are necessary)
    • Get an outside opinion! Discuss the idea with your reviewer first and show them your mockups and diagrams

    Sometimes I write the PR description before I even start coding. This keeps the scope more focussed, and I can catch myself when I start to veer off-topic.

    I still don’t always get it right — sometimes my PRs are too big, or my explanations too vague. But thinking intentionally about the process has helped me a lot. Code review is more than just checking for bugs, it’s about building shared understanding. By planning better, scoping changes thoughtfully, and writing clear PRs, we make collaboration smoother for everyone. It takes practice, but the payoff is real: better software, better teams.

    There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.

    • C.A.R. Hoare
  • No abstraction is better than the wrong abstraction

    The DRY principle is a widely accepted guideline for writing maintainable and readable code. As described by David Thomas and Andrew Hunt in the Pragmatic Programmer:

    Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

    This statement is often boiled down to: “Don’t repeat yourself”. This is an essential heuristic for writing great code (no one wants to have to fix a bug by having to apply the same fix in 10+ different locations in the codebase!), but there’s some important nuance that is often missed when applying this advice dogmatically.

    Coding principles and advice like this are incredibly valuable, but they’re often written from the perspective of “what the finished product should look like”. As a consequence, explanations and discussions therein miss crucial details of the messy process on the journey to the finished product – and that’s what more people often need help with!

    “Don’t repeat yourself” is unfortunately interpreted as “create an abstraction before you have a chance to copy-paste code”, such as a new function or class for this particular snippet. But there are many pitfalls to this approach:

    • You haven’t taken the time to observe more than one or two use cases for this logic, how can you be more than 50% certain that the interface is what you’ll need in the future?

    • The requirements and responsibilities of this abstraction will likely change as you get deeper into solving the problem for which you’re writing the code for.

    • Premature abstraction can lead to a greater sin: non-orthogonal architecture and heavy coupling.

    Designing code for future use is another pitfall, but that’s a topic for another post. In my observation so far, the wrong abstraction is often hastily modified as new requirements come in, and this can become baked into the codebase. Working around this or changing it is often a much harder task than pulling a few duplicated parts later down the line.

    The advice given by Sandi Metz states a great pragmatic solution for dealing with this:

    If you find yourself in this situation, resist being driven by sunk costs. When dealing with the wrong abstraction, the fastest way forward is back. Do the following:

    1. Re-introduce duplication by inlining the abstracted code back into
       every caller.
    2. Within each caller, use the parameters being passed to determine the
       subset of the inlined code that this specific caller executes.
    3. Delete the bits that aren't needed for this particular caller.

    This advice makes me think of another proverb which I’ve found useful:

    No matter how far down the wrong road you have gone, turn back now!

    Re-implementing the duplication (or writing the duplication in the first place) lets a better abstraction coalesce: You have more context, and hopefully a better idea of the requirements and flow of logic in the system you’re trying to design.

    Duplication is a natural part of the writing process. You will never write the perfect version on the first attempt (probably not even on the 10th attempt). But you can iterate on your messy code that works and improve it! So embrace it — Get your thoughts on the page. Make it work, then make it maintainable (then make it fast!). Once a working version is implemented, refactor and rewrite parts where necessary.

    Once you’ve had a chance to work through the problem a few times, the right abstractions are more likely to jump out at you as you review what you’ve written.

  • AI workforce reconsidered, my response

    Recently someone sent me this article from futurism where the fintech company Klarna is reversing course on their drive to replace a large portion of their marketing workforce with AI over the last two years.

    While this move saved costs, it led to a noticeable drop in service quality. As a result, they’re planning to bring humans back into customer support roles. I think this wont be the last we see article like this, as the AI hype starts to settle and the limitations become more apparent.

    shocked_lol

    Memes aside, I’ve always suspected this drive some vocal CEOs are embarking on to replace people with AI will be a short-term gain with long-term pain. Klarna is reconstructing their customer-facing roles, but I fear their consequences would be much more substantial if they had bought into replacing technical roles (such as software engineering) with AI.

    LLMs and AI agents are incredible tools with so much capability, I don’t think this should be downplayed at all. But that being said, they are also great at producing the most subtle bugs and issues that require a lot of skill to fix – sort of like a photocopy of a photocopy.

    https://graphicdesign.stackexchange.com/questions/82081/reasons-of-loss-in-image-quality-in-subsequent-photocopy

    These models seem magical, but they’re ultimately only replicating patterns they’ve been exposed to in their training data. A lot of the time, that’s is very useful, but when used irresponsibly (or for decision-making) it can be pretty difficult to fix the mess it creates.

    These subtle issues compound because a lot of life is non-linear and chaotic like a double-pendulum. A subtle difference can have compounding impacts and outcomes way down the line.

    https://drwealth.com/double-pendulums-and-chaotic-vs-random-market-behaviour/

    Personally, I reckon CEOs that jumped on this bandwagon reap what they sowed. I’m more saddened by the people who are caught in the cross-fire and lost their jobs as a result of this poor decision making.