I’m Blake. I write about probability, stochastic processes, Python tooling, and the practical side of building things with code. More about me →

  • TIL: How to parse config from env vars in Python

    In an effort to collate all my runtime options and parameters into a single source of truth, I initially created a src/globals.py file with the values and their defaults:

    import os
    
    FOO: str = os.get("FOO", "my_value")
    BAR: int = int(os.get("BAR", "55"))
    BAZ: bool = os.get("BAZ", "False").lower().startswith("t")

    A simple solution, these symbols can then be imported by using from src.globals import FOO, ..., but this solution has a couple of drawbacks:

    • Values are set only once when the import statement is executed
    • Values can’t be overridden in Python without creating a bit more noise

    For example, let’s say we have a function floob that uses a dict containing foo, bar, and baz:

    from src.globals import FOO, BAR, BAZ
    
    
    def floob(data: dict[str, str | int | bool] | None = None):
        data = data or {}
        foo = data.get("foo", FOO)
        bar = data.get("bar", BAR)
        baz = data.get("baz", BAZ)
        ...

    Admittedly this is a contrived example, but it can start to get unwieldy as the number of get statements increases and becomes more scattered about. Not the mention there is the possibility of a bug in globals.py when casting items to their correct types.

    I came across pydantic_settings as a solution to this by providing a class to configure this.

    from pydantic import Field
    from pydantic_settings import BaseSettings
    
    
    # An env prefix can optionally be specified as well
    class MyConfig(BaseSettings, env_prefix="APP_"):
        foo: str = Field(
            description="foo",
            default="my_value",
        )
        bar: int = Field(
            description="bar",
            default=55,
        )
        baz: bool = Field(
            description="baz",
            default=False,
        )

    When this class is instantiated, the values are looked up from the given kwargs, then dynamically looked up from the environment, and then falls back to the defaults:

    # $ export APP_FOO="env_value"
    config = Config(baz=True)
    
    print(config.foo)
    # "env_value"
    print(config.bar)
    # 55
    print(config.baz)
    # True

    Where this really shines is in the integration with the rest of pydantic, such as value validation:

    from typing import Annotated
    
    from pydantic import Field, AfterValidator
    from pydantic_settings import BaseSettings
    
    
    def is_allowed(value: str) -> str:
        if value is in ["not_allowed", "wumbo"]:
            raise ValueError(f"Invalid string value: '{value}'")
        return value
    
    CheckedStr = Annotated[str, AfterValidator(is_allowed)]
    
    # An env prefix can optionally be specified as well
    class MyConfig(BaseSettings, env_prefix="APP_"):
        foo: Checked = Field(
            description="foo",
            default="my_value",
        )
        bar: int = Field(
            description="bar",
            default=55,
        )
        baz: bool = Field(
            description="baz",
            default=False,
        )
    
    # $ export APP_FOO=wumbo
    _ = Config()  # Raises a validation error

    The end result is a more flexible config class, cleaner definition of defaults, robust parsing of env vars without the standard boilerplate, on top of the validation and type-checking from pydantic. …

  • Bug hunt: Rescaling DataFrames in Python

    This is one of the tricky bugs I fixed relatively early in my career. This one took a solid few days to truly understand, but it boiled down to a relatively simple fix.

    The problem

    This bug affected a data export pipeline in a legacy codebase. We were converting raw time series values into a proprietary data format (with permission) so that the data could be inspected in another program by the technicians.

    Some of the technicians raised a bug report to us:

    Some of the files would crash the program when opened

    Their workaround was to re-create the file on a subset of the data – indicating there was potentially some bad values that weren’t being scaled properly, probably leading to an “out-of-bounds” error.

    The steps that are taken to convert the raw data:

    • Clip values to +/- 5 mV, which are the limits of what can be represented in the proprietary filetype
    • Rescale the channels so that all values are within +/- 512, which is the range of integers that can be encoded in 10 bits
    • Split up the study data into 1 hour frames
    • Convert each scaled frame of data into binary and append to proprietary file.

    The part of the pipeline concerned with rescaling values was

    import pandas as pd
    
    
    def rescale_channels(
        data: pd.DataFrame,
        channel_headers: list[str],
    ) -> pd.DataFrame:
        maximum_point = data[channel_headers].max().max()
        minimum_point = data[channel_headers].min().min()
        abs_max = max(maximum_point, abs(minimum_point))
    
        absolute_max = 5 * (10**-3)
        for column in channel_headers:
            _data = data[column]
    
            # CLIPPING
            if abs_max > absolute_max:
                abs_max = 5 * (10**-3)
                _data = _data.clip(lower=-abs_max, upper=abs_max)
    
            # RESCALING
            _data = (_data / abs_max * 510).round()
            _data = _data.fillna(511)  # 511 is "undefined".
    
            # TYPECASTING
            _data = _data.astype(int)
    
            # RECOMBINING COLUMN IN DF
            data[column] = _data
    
        return data

    Can you spot the bug?

    The solution

    Turns out the issue was an overwritten loop variable:

    # CLIPPING
    if abs_max > absolute_max:
        abs_max = 5 * (10**-3)  # <--- ISSUE OCCURS HERE
        _data = _data.clip(lower=-abs_max, upper=abs_max)

    In cases when 5mV < absmax(CH1) < absmax(CH2), the highlighted if statement is only entered once, resulting in values in CH2 that exceed +/- 512.

    The fix is thankfully quite simple:

    # channel_headers = ["ECG1", "ECG2", "DIFF"]
    def rescale_channels(
        data: pd.DataFrame,
        channel_headers: list[str],
    ) -> pd.DataFrame:
        maximum_point = data[channel_headers].max().max()
        minimum_point = data[channel_headers].min().min()
        abs_max = max(maximum_point, abs(minimum_point))
    
        absolute_max = 5 * (10**-3)
        for column in channel_headers:
            abs_max_col = (
                abs_max  # <--- CREATE A TEMP ITERATION VARIABLE TO AVOID OVERWRITE
            )
            _data = data[column]
    
            # CLIPPING
            if abs_max_col > absolute_max:
                abs_max_col = 5 * (10**-3)
                _data = _data.clip(lower=-abs_max_col, upper=abs_max_col)
    
            # RESCALING
            _data = (_data / abs_max_col * 510).round()
            _data = _data.fillna(511)  # 511 is "undefined".
    
            # TYPECASTING
            _data = _data.astype(int)
    
            # RECOMBINING COLUMN IN DF
            data[column] = _data
    
        return raw_data

    Once this was implemented, I added a unit test to capture this case and ensured that the issue was fixed when the technicians opened their files with this fix implemented.

    Lesson I learned from this:

    • Don’t overwrite loop variables – this leads to tricky to spot bugs!
  • TIL How to compose functions and coroutines in Python

    A list of functions can be composed using this pattern:

    # Some example functions
    def foo(x):
        return 3 * x
    
    
    def bar(x):
        return x + 3
    
    
    def baz(x):
        return x**3
    
    
    # Function to compose functions
    def compose(*funcs):
        def composition(x):
            for func in funcs:
                x = func(x)
            return x
    
        return composition
    
    
    # Example usage
    def main():
        pipeline = compose(foo, bar, baz)
    
        x = 5
        x_out = pipeline(x)
        print(x_out)  # 5832
    
    
    if __name__ == "__main__":
        main()

    But what if one of your functions is a coroutine? And another one as a class with an asynchronous __call__ method?

    import asyncio
    
    
    async def foo_async(x):
        await asyncio.sleep(1)
        return 3 * x
    
    
    class BarAsync:
        def __init__(self, floob):
            self.floob = floob
    
        async def __call__(self, x):
            await asyncio.sleep(self.floob)
            return 3 * x
    
    
    def baz(x):
        return x**3

    Well we can modify our pipeline to handle these cases:

    def compose_async(*funcs):
        async def composition_async(x):
            for func in funcs:
                result = func(x)
                if asyncio.iscoroutine(result):
                    x = await result
                else:
                    x = result
            return x
    
        return composition_async
    
    
    def main():
        pipeline = compose_async(foo_async, BarAsync(2), baz)
    
        x = 5
        x_out = asyncio.run(pipeline(x))  # or `await` this in another coroutine
        print(x_out)  # 5832
    
    
    if __name__ == "__main__":
        main()

    Bonus tip: here’s how to type-hint the example to keep mypy happy:

    import asyncio
    from typing import Union, Callable, Awaitable, cast
    
    
    # Pipeline functions
    async def foo_async(x: int) -> int:
        await asyncio.sleep(1)
        return 3 * x
    
    
    class BarAsync:
        def __init__(self, floob: int):
            self.floob = floob
    
        async def __call__(self, x: int) -> int:
            await asyncio.sleep(self.floob)
            return 3 * x
    
    
    def baz(x: int) -> int:
        return x**3
    
    
    # Define a type alias for functions in the pipeline
    PipelineFunc = Union[Callable[[int], int], Callable[[int], Awaitable[int]]]
    
    
    # Define pipeline
    def compose_async(*funcs: PipelineFunc) -> Callable[[int], Awaitable[int]]:
        async def composition_async(x: int) -> int:
            for func in funcs:
                result = func(x)
                if asyncio.iscoroutine(result):
                    x = await result
                else:
                    x = cast(int, result)  # Tell mypy explicitly about type
            return x
    
        return composition_async
    
    
    # Example of pipeline usage
    def main() -> None:
        pipeline = compose_async(foo_async, BarAsync(2), baz)
    
        x = 5
        x_out = asyncio.run(pipeline(x))  # or `await` this in another coroutine
        print(x_out)  # 5832
    
    
    if __name__ == "__main__":
        main()
  • Tuner: why dating apps (probably) don't match with Spotify data

    When I was on Tinder many years ago, the app prompted me to enter my Spotify details to my profile. When I did, some of my top artists and albums appeared on my profile which seemed a bit lackluster. This made me think of the compatibility of people based on their taste in music, and I wondered: why isn’t this data being used to help match people?

    At the time I didn’t have the skills or the tools needed to explore this idea. But as I outlined in my post a couple of months ago, I was able to combine newly acquired skills with Flask and embeddings to create a proof-of-concept of this idea: tuner.

    Using the Spotify API was a relatively straightforward process: sign into https://developer.spotify.com/, create an app, enter in the redirect URL, and copy your Client ID and secret ID. This grants you developer privileges, meaning your account has access to your data and public data, but not anyone else’s data.

    In order to get permissions for other peoples data, you need to submit a request to Spotify via their developer portal with a brief description of what your app is and what it does (plus a demo video). This sounded like a great idea, I could open up this proof-of-concept to the world and let people learn about embeddings using their music data!

    So I applied, and got a response after a month or so:

    Dear Tuner team,

    Thank you for submitting Tuner to Spotify for review.

    We appreciate your interest and efforts in using Spotify’s open platform to innovate and build interesting integrations. However, after reviewing we found that your app does not comply with our terms and conditions for the following reasons:

    * You must not analyze the Spotify Content or the Spotify Service for any
      purpose, including without limitation, benchmarking, functionality,
      usage statistics, or user metrics
    * The product or service is integrated with streams or content from
      another service

    We will therefore not be able to support your use case.

    An unfortunate outcome… Addressing their points in reverse order:

    1. The only reason I needed to source data from other services is because they changed the API halfway through my project, and made it impossible to get related tracks and small track snippets when generating a recommended playlist.

    2. The definition of “analyzing Spotify Content” is a bit broad, and perhaps so is the definition of “user metrics”, but I would argue I’m not analysing the data:

      • This app uses the genre tags of a users top artists to determine their music taste, this is the extent of Spotify data that’s used
      • There is no analysis of data taking place – only computation of a users tags to generate Spotify playlists

    The genre vector computed from the embedding is a different representation of their data which isn’t analysed. But taking the definition of a metric as “a noun describing a specific type of measurement”, I can see how they arrived at their interpretation.

    How does this project conclude? I’m tempted to revisit the app and replace the extra service data access with placeholders. But addressing the “analyzing data” clause will be much more difficult. Perhaps there’s a way to fuzz the data so it’s not matching users? I think there’s still utility in this to generate playlists to help users discover new artists, but it seems debatable if that would even be allowed.

    In any case, publishing this proof-of-concept was an extension of my original project. I wanted to answer “why isn’t Spotify data being used to help match people?”, and I can conclude the reason is not because the technology isn’t there, but because Spotify’s terms of usage might not allow it.

    (Anyway, I’ll be keeping an eye out to see if Spotify releases a matching algorithm or partners with a dating app)