I’m Blake. I write about probability, stochastic processes, Python tooling, and the practical side of building things with code. More about me →

  • Old Hat, New Tricks: Makefiles for Python Projects

    Makefiles have been a staple of the C/C++ toolchain for decades. In a nutshell: GNU Make executes discrete steps that are defined in a Makefile. The steps defined in a Makefile can have dependencies, so that when a step is executed, all the dependent steps are executed first. This enables effective version control and cleaner documentation for the manual steps that are needed to build and run a project.

    Like many GNU utilities, it’s been around for a while and is easily overlooked by developers until they encounter a project that uses them effectively. I recently had that experience in some recent Python repositories I’ve been working in. At first I was skeptical, but there are many benefits to using them in modern projects that I didn’t previously consider:

    • Less documentation to write and maintain
      • Dependencies between steps are built-in to the Makefile
      • Just a brief summary of Makefile steps for your project will suffice

    Previously I used a stack of bash scripts to achieve the same goal. But I found that there’s a lot of careful organisation needed to avoid user error caused by executing steps in an incorrect order. Now I find that with using the Makefile, much shorter documentation is needed – as all the details and the dependencies between steps are handled in the construction of the Makefile. The steps in make can also execute any shell script, so steps can still be written in scripts and orchestrated with make.

    One sneaky “gotcha” with writing a Makefile is that the steps require indentation with one tab character, not spaces (I found this blog post, which contains an email from the author of make who regrets this decision with good humor!). I use this syntax file to handle this automatically when writing a Makefile in vim:

    " ~/.vim/ftplugin/make.vim
    set noexpandtab shiftwidth=8 softtabstop=0

    So why is it useful in Python? There are so many ways to manage dependencies in Python (and of course, none of them quite meet the 100% mark). Now instead of meticulously documenting how to use the tool of choice for a project, I simply document which tool is needed and let the Makefile contain all the details.

    For example, here’s a Makefile I would write for a Python project that uses poetry for dependency management, pytest for testing, ruff for linting and formatting, and docker for building images:

    SHELL := /usr/bin/env bash
    .SHELLFLAGS := -eu -o pipefail -c
    
    
    .PHONY: _check-python-version, install-uv, install-py-deps, install-py-deps-dev, check-tests, check-lint, check-format, check-all, build-image, build-image-dev, load-dev-shell
    
    
    _check-python-version:
        @version=$$(python --version 2>&1); \
            if [ "$$version" != "Python 3.13.7" ]; then \
            echo "Error: Python version is not 3.12.7. Detected version: $$version"; \
            exit 1; \
        fi
    
    install-poetry: _check-python-version
        @brew install pipx
        @pipx install poetry==1.8.4 --python=$(which python)
    
    install-py-deps:
        @poetry install --sync --only=main
    
    install-py-deps-dev:
        @poetry install --sync --only=main,dev
    
    check-lint:
        @poetry run pytest
    
    check-lint:
        @poetry run ruff check
    
    check-format:
        @poetry run ruff format --check .
    
    check-all: check-tests check-lint check-format
    
    build-image-base:
        @docker build -t "my-container-base" -f Dockerfile.base .
    
    build-image-dev: build-image
        @docker build -t "my-container-dev" -f Dockerfile.dev .
    
    load-dev-shell: build-dev-image
        @docker exec -it "my-container-dev" bash
    

    The targets in .PHONY ensure that the steps execute every time they’re called, and @ suppresses the command output. Using this file, all my steps are version controlled, unwieldy flags and options for commands don’t need to be copy-pasted, and users with brew can get up and running with a few simple commands:

    $ make install-poetry
    $ make install-py-deps-dev
    $ make check-all
    $ make load-dev-shell

    The Makefile is here to stay. It’s simplified my development process, and it’s now one of the first things I’ll write for a new project

    References:

  • TIL How to add a progress wheel to a blocking Python function

    Threads can be used to provide a progress wheel for long-running python functions

    import concurrent.futures as cf
    import sys
    import threading
    import time
    
    
    def slow_function(x: int | float) -> str:
        time.sleep(x)
        return f"Operation took {x} seconds"
    
    
    def print_progress(stop_event):
        # Create a spinning progress indicator
        spin_chars = ["⣾", "⣽", "⣻", "⢿", "⡿", "⣟", "⣯", "⣷"]
        while not stop_event.is_set():
            for char in spin_chars:
                if stop_event.is_set():  # Check if we need to stop the thread
                    return
                sys.stdout.write(f"\r{char} Waiting...")
                sys.stdout.flush()
                time.sleep(0.1)
    
    
    def main():
        stop_event = threading.Event()  # Create an event to signal thread to exit
        with cf.ThreadPoolExecutor() as executor:
            future_spinner = executor.submit(print_progress, stop_event)
            try:
                result = slow_function(2)
            except KeyboardInterrupt as err:
                raise err
            finally:
                stop_event.set()
                cf.wait([future_spinner])
                sys.stdout.write("\r\033[K")  # Clear the current line
    
        return result
    
    
    if __name__ == "__main__":
        try:
            print(main())
        except KeyboardInterrupt:
            print("Operation cancelled")

    This pattern seems like a good candidate for a decorator:

    import concurrent.futures as cf
    import sys
    import threading
    import time
    
    
    def progress_wheel(func):
        def print_progress(stop_event):
            # Create a spinning progress indicator
            spin_chars = ["⣾", "⣽", "⣻", "⢿", "⡿", "⣟", "⣯", "⣷"]
            while not stop_event.is_set():
                for char in spin_chars:
                    if stop_event.is_set():  # Check if we need to stop the thread
                        return
                    sys.stdout.write(f"\r{char} Waiting...")
                    sys.stdout.flush()
                    time.sleep(0.1)
    
        def wrapped_func(*args, **kwargs):
            stop_event = threading.Event()  # Create an event to signal thread to exit
            with cf.ThreadPoolExecutor() as executor:
                future_spinner = executor.submit(print_progress, stop_event)
                try:
                    result = func(*args, **kwargs)
                except KeyboardInterrupt as err:
                    raise err
                finally:
                    stop_event.set()
                    cf.wait([future_spinner])
                    sys.stdout.write("\r\033[K")  # Clear the current line
    
            return result
    
        return wrapped_func
    
    
    @progress_wheel
    def slow_function(x: int | float) -> str:
        time.sleep(x)
        return f"Operation took {x} seconds"
    
    
    def main():
        return slow_function(2)
    
    
    if __name__ == "__main__":
        try:
            print(main())
        except KeyboardInterrupt:
            print("Operation cancelled")

    Experiment with other progress displays, I found a few great ideas using unicode from this stack overflow thread

  • Tuner: A web app for finding people with similar music tastes

    Tuner is a web app I’ve been working on the side for the past 3 months. Many years ago, I wanted to use Spotify data to match people with similar music tastes, and improve discovery of new artists but using connections with other people’s music tastes. Essentially, this is how Spotify works under the hood (at least as far as I’m aware). The recommendation systems leverages the listening activity of all users in order to make suggestions.

    But I wanted to add a more personal touch to these recommendations. A majority of my music library has come from artists I’ve discovered through my friends. I wondered about how this process could be done over the web, but it seemed to be out of reach.

    That is, until I learned about vector databases.

    After pursuing another flask project, I learned enough about web hosting and flask to finally put the pieces together

    How it works

    I learned about embeddings at the start of the year, but didn’t really understand then until I started getting my hands dirty in a new job.

    Embeddings are a neutral network model that provide a way to quantify semantic information. First a string input is tokenized and mapped to a sequence of numbers, then this is embedded into a high-dimensional vector space. Word2vec is one of the earlier embeddings that gained a lot of attention. They now are used pretty widely in natural language processing.

    For this project I used an off the shelf embedding from sentence_transformers (all-MiniLM-L6-v2) to embed music genres in a vector space.

    Tuner loads the top 20 artists for a user, then records the genres each artist is tagged as. Each of these genres can be embedded into the vector space, and these vectors can be added together to form a “music taste” vector for each user. Then by calculating the similarity of vectors (by measuring the cosine of the angle between them), users with similar music tastes can be matched and ranked.

    What it was like building it

    It was a very fun experience building this tool. It started off as a command line interface in a basic prototype, then eventually became a flask app.

    There were some interesting challenges regarding speed that had to be solved. One major speedup was achieved by pre-computing the embeddings for the genres, then looking up the vectors in a table instead of running the model on the backend.

    I also had to think more about making this a more enjoyable tool to use, so I decided to use the Spotify API to recommend a playlist for the user. This playlist would be based on what artists were common for user a and user b, as well as artists that user b liked that user a didn’t have listed in their top artists.

    What was the setback?

    Frustratingly, Spotify announced suddenly that some of their endpoints would be closed effectively immediately. This announcement came out on Thanksgiving, and had no prior indication that it was coming. While it was frustrating for me, it was heartbreaking to see other developers stories of their tools and scripts suddenly cease to work as I trawled through the developer support forums.

    Thankfully, this only broke the playlist generation part of my web app, not the core concept. I spent a week on the playlist feature, and learned a lot of basic html and JavaScript to make it work, so I was determined to salvage it.

    I came up with a basic solution that would query the LastFM API for related artists and songs, but this meant I had to individually match each song to a URI on Spotify in order to create a playlist for the user. This solution took over 40 seconds to load a playlist, which is unacceptably slow for a web page.

    So i got my hands dirty with multithreading, and improved the speed of this playlist generation to around 5 seconds (once I factored in rate limits for the respective APIs). I might write a more detailed post on how I used asyncio in Python to achieve this in the future.

    What I learnt

    I learned some incredible ideas and skills in basic web design and backend processing. I learned about the basics of OAuth and how API authentication works. JavaScript is a language I haven’t spent much time in, so the front end was particularly challenging.

    This was a great opportunity to learn about embeddings, and it was a joy to bring a far off idea from several years ago to life.

    What I want to add to it

    When I return to the project, I want to learn about fine tuning embeddings. For now, a pertained model is sufficient, and a fine tuned model will only produce a marginal gain. But it seems like a good way to dive deeper into the world of embeddings and natural language processing

    So is it a good recommendation system? Honestly, probably not better than conventional approaches. There’s a more human element to these recommendations, but the quality and range of matches depends on how many people engage with the service since I can only match people who have given permission to tuner. Initially I’ve filled the database with mock data based on popular public playlists so early users can still experience tuner as it was meant to be experienced. But as more users use tuner, I’ll be able to remove these entries. Tuner is an open source project, feel free to check out the repo and get in touch if you have any questions or comments!

    Many thanks to the developers of spotipy, sentence_transformers, pinecone, flask, render, everynoise, and pylast. This fun little project would not have been possible without all the contributions made to these projects!

  • TIL How to secure a web server with SSO

    I was setting up an instance with the clearml-server for my team to start tracking their data science experiments and versioning their datasets and tracing artifacts. They were impressed with the application, however they raised some valid concerns:

    Is it secure? Can we add SSO to this server?

    I scoured the docs, apparently it’s available in the enterprise edition, but I was more keen to self-host and learn more about the process.

    The details don’t really matter that much, but in a nutshell:

    • Hosted on Port 5050
    • Orchestrated with Docker Compose

    Turns out nginx might provide a solution! I came across a container for oauth2-proxy, which can be placed in front of my web app by nginx. In short, nginx directs traffic to oauth2-proxy, which then directs traffic to an underlying web server after a successful authentication.

    On my server, I created an nginx directory:

    /opt/nginx/
    ├── certs
    │   ├── tailscaled.crt
    │   └── tailscaled.key
    └── nginx.conf

    I’m using tailscale as my VPN solution for this project, which gives me a domain name that can only be accessed on my network, and also can issue a signed SSL certificate when you enable HTTPS and Magic DNS on your network. The certificate can be generated with tailscale cert <domain-name>.

    The content of nginx.conf is:

    worker_processes 1;
    
    events {}
    
    http {
        log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
    
        access_log /var/log/nginx/access.log main;
    
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
    
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
    
        # HTTPS server block
        server {
            listen 443 ssl;
    
            # Use Tailscale-provided SSL certificates
            ssl_certificate /etc/nginx/certs/tailscaled.crt;
            ssl_certificate_key /etc/nginx/certs/tailscaled.key;
    
            # Proxy traffic to oauth2-proxy
            location / {
                proxy_pass http://oauth2-proxy:4180;  # Ensure this matches the oauth2-proxy service
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
            }
        }
    
        # Optional HTTP-to-HTTPS redirect block
        server {
            listen 80;
    
            return 301 https://$host$request_uri;
        }
    }

    If HTTPS isn’t required, the content of the server block after listen 443 ssl; can be moved into the server { listen 80; ... } block and the return 301 can be removed as well. Also ignore the ssl_certificate and ssl_certificate_key.

    I then used this docker-compose.yml to spin up nginx and oauth2-proxy:

    version: "3.6"
    services:
      nginx:
        image: nginx:latest
        container_name: nginx
        volumes:
          - /opt/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
          - /opt/nginx/certs:/etc/nginx/certs:ro
        ports:
          - "80:80"
          - "443:443"
        depends_on:
          - oauth2-proxy
        networks:
          - frontend
    
      oauth2-proxy:
        image: quay.io/oauth2-proxy/oauth2-proxy:latest
        container_name: oauth2-proxy
        environment:
          - OAUTH2_PROXY_CLIENT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
          - OAUTH2_PROXY_CLIENT_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
          - OAUTH2_PROXY_REDIRECT_URL=https://my-webserver-name.tailnet1234.ts.net/oauth2/callback
          - OAUTH2_PROXY_COOKIE_SECRET=XXXXXXXXXXXXXXXX
          - OAUTH2_PROXY_PROVIDER=google  # Or another OAuth provider
          - OAUTH2_PROXY_UPSTREAMS=http://my-webserver-container:5050
          - OAUTH2_PROXY_EMAIL_DOMAINS=company.com
          - OAUTH2_PROXY_HTTP_ADDRESS=0.0.0.0:4180  # Default OAuth2 proxy port
          - OAUTH2_PROXY_SKIP_PROVIDER_BUTTON=true  # Disable the OAuth2 provider button if needed
          - OAUTH2_PROXY_COOKIE_SECURE=true  # Enable if serving through HTTPS
          - OAUTH2_PROXY_COOKIE_HTTPONLY=true  # Use HTTP-only cookies for added security
        ports:
          - "4180:4180"
        networks:
          - frontend
        restart: always

    Now with all the containers running, I’m able to navigate to https://my-webserver-name.tailnet1234.ts.net/oauth2/callback in my browser when connected to my VPN. I’m immediately redirected to an SSO page (from Google in this particular case). Once I sign in and confirm my identity. I get redirected back to the domain and I’m able to access the server web interface!

    Also make sure to close the port for the underlying web app ;)