I’m Blake. I write about probability, stochastic processes, Python tooling, and the practical side of building things with code. More about me →

  • Real-Time Voice Streaming and Stateful Buffering

    Last year I wanted to build a live transcriber/fact extractor to learn more about the process of building such a system. To my surprise, downloading and using a Whisper model in Python is pretty straightforward

    But like with any side project — the hard part is always something unexpected. In this case, it was streaming audio processing. I manged to get something working last year, but honestly it was a hot mess of threads and queues. This week I revisited this repo and revised my approach (and open sourced it here!).

    The challenge with streaming audio processing is knowing when to start listening and when to stop. The Whisper model has some VAD (voice activity detection) built in, but I needed to reduce load on the CPU. Sending any chunks of audio would slow the system down, so I needed to minimise unnecessary calls to the model.

    So here is a walkthrough of the buffering solution I used that could detect arbitrary speech onset, capture the full utterance with natural padding, and handle the messy reality of human speech (pauses, false starts, and varying speech lengths).

    Architecture Overview

    The solution splits into two main components:

    1. VADAudioBuffer: A stateful buffer that implements the “listening logic” using WebRTC’s Voice Activity Detector
    2. VoiceStreamer: A threading wrapper that bridges audio I/O (via sounddevice) with your processing pipeline

    The Buffering Logic: Lookback, Trigger, and Padding

    The heart of the system is the VADAudioBuffer.process() method, which implements a state machine with three distinct phases:

    Phase 1: The Lookback Ring Buffer (Pre-trigger)

    Before any speech is detected, audio flows into a ring buffer (self.ring_buffer) that stores the last 300ms of audio (configurable via padding_duration_s). This is crucial because human speech doesn’t start abruptly at the detection threshold. There are often breaths, plosives, or soft beginnings that the VAD might initially miss as “non-speech.”

    # Simplified logic
    self.ring_buffer.append((frame, is_speech))
    if not self.triggered:
        num_voiced = sum(1 for _, s in self.ring_buffer if s)
        if num_voiced > 0.9 * self.ring_buffer.maxlen:
            self.trigger()  # Captures ring_buffer contents + new frames

    Phase 2: Active Recording (Triggered State)

    Once triggered, the buffer switches modes. Instead of overwriting old data, it accumulates frames in self.voiced_frames. The system stays in this state until it detects sustained silence—specifically, when 90% of the ring buffer contains non-speech frames.

    This is where the max_duration_s safety limit becomes important. Without it, a continuous noise (air conditioning, road noise mistaken for speech) could accumulate indefinitely. The system forces a flush when max_frames is reached.

    Phase 3: Post-Processing and Padding

    When silence is detected (or max duration hit), flush_buffer() combines all accumulated frames into a single byte string, converts it from int16 back to float32, and returns the numpy array.

    Notice that the final utterance includes:

    1. The lookback frames (captured speech onset)
    2. All voiced frames during the triggered state
    3. Natural padding at the end (the silence that triggered the stop)

    The Streaming Layer: Threading and Generators

    The VoiceStreamer class solves the I/O blocking problem. Audio capture happens in a callback thread (via sounddevice), but VAD processing happens in a dedicated worker thread. This separation prevents audio dropouts if your processing callback (e.g., speech-to-text) takes variable time.

    The class implements the context manager protocol and yields results as a generator:

    with VoiceStreamer(record_device=0, callback=my_stt_function) as streamer:
        for result in streamer:
            print(result)

    Writing this with asyncio instead of generators is definitely an option, but I was already pretty far into unexplored territory. I didn’t want to get too lost in the weeds before getting something working!

    Edge Cases and Tuning

    This architecture handles several real-world scenarios gracefully:

    • False starts: If someone coughs or a chair scrapes, the lookback buffer prevents the system from capturing only the tail end of the noise. The 90% threshold requires sustained speech.
    • Pause in speech: The padding duration (300ms) allows for natural pauses between words. If the pause exceeds the padding, the buffer flushes, and a new utterance starts when speech resumes.
    • Continuous noise: The max duration limit prevents memory issues.

    The Complete Picture

    import collections
    import logging
    import queue
    import threading
    from typing import Any, Callable
    
    import numpy as np
    import sounddevice as sd
    import webrtcvad
    
    logger = logging.getLogger(__name__)
    
    
    class VADAudioBuffer:
        sample_rate = 16000
        frame_duration_s = 0.03
    
        def __init__(
            self,
            padding_duration_s=0.3,
            lookback_s=0.5,
            max_duration_s=5.0,
            min_duration_s=0.3,
        ):
            self.padding_duration_s = padding_duration_s
            self.lookback_s = lookback_s
            self.max_duration_s = max_duration_s
            self.min_duration_s = min_duration_s
    
            self.frame_size = int(self.sample_rate * self.frame_duration_s)
            self.lookback_frames = int(self.lookback_s / self.frame_duration_s)
            self.padding_frames = int(self.padding_duration_s / self.frame_duration_s)
            self.max_frames = int(self.max_duration_s / self.frame_duration_s)
            self.min_frames = int(self.min_duration_s / self.frame_duration_s)
    
            self.vad = webrtcvad.Vad(1)
    
            self.triggered = False
            self.voiced_frames = []
            self.ring_buffer = collections.deque(maxlen=self.padding_frames)
    
        def process(self, data: np.ndarray) -> np.ndarray | None:
            audio_np = None
            frame_size_bytes = self.frame_size * 2  # 2 bytes per int16 sample
            if len(data) % frame_size_bytes != 0:
                raise ValueError(
                    f"Chunk must be a mutliple of {frame_size_bytes} samples "
                    f"(got {len(data)})"
                )
    
            # Convert float32 audio to int16 for VAD processing
            # data is mono float32 from the stream, flatten in case it's (N, 1)
            data_flat = data.flatten()
            chunk = (data_flat * 32767).astype(np.int16).tobytes()
    
            for i in range(0, len(chunk), frame_size_bytes):
                frame = chunk[i : i + frame_size_bytes]
                is_speech = self.vad.is_speech(frame, self.sample_rate)
    
                self.ring_buffer.append((frame, is_speech))
    
                if not self.triggered:
                    num_voiced = sum(1 for _, s in self.ring_buffer if s)
    
                    if num_voiced > 0.9 * self.ring_buffer.maxlen:
                        logger.debug("Triggered")
                        self.load_buffer()
    
                else:
                    self.voiced_frames.append(frame)
                    num_unvoiced = sum(1 for _, s in self.ring_buffer if not s)
    
                    # Force flush when max duration is reached
                    if len(self.voiced_frames) >= self.max_frames:
                        audio_np = self.flush_buffer()
    
                    elif num_unvoiced > 0.9 * self.ring_buffer.maxlen:
                        audio_np = self.flush_buffer()
    
                        self.unload_buffer()
    
            return audio_np
    
        def post_process(self, utterance: bytes) -> np.ndarray:
            audio_np = np.frombuffer(utterance, dtype=np.int16)
            audio_np = audio_np.astype(np.float32)
            audio_np /= 32768.0
            return audio_np
    
        def unload_buffer(self):
            logger.debug("Untriggered")
            self.triggered = False
            self.ring_buffer.clear()
    
        def load_buffer(self):
            logger.debug("Triggered")
            self.triggered = True
            self.voiced_frames = [f for f, _ in self.ring_buffer]
            self.ring_buffer.clear()
    
        def flush_buffer(self) -> np.ndarray:
            if len(self.voiced_frames) <= self.min_frames:
                return None
    
            utterance = b"".join(self.voiced_frames)
            audio_np = self.post_process(utterance)
    
            self.voiced_frames = []
            return audio_np
    
    
    class VoiceStreamer:
        model_size = "small.en"
    
        def __init__(
            self,
            record_device: int,
            callback: Callable[[np.ndarray], Any | None] | None,
        ):
            self.record_device = record_device
            self.callback = callback
    
            self.vad_buffer = VADAudioBuffer()
    
            self.input_stream = sd.InputStream(
                samplerate=self.vad_buffer.sample_rate,
                blocksize=2
                * int(self.vad_buffer.sample_rate * self.vad_buffer.frame_duration_s),
                dtype=np.float32,
                channels=1,
                device=self.record_device,
                callback=self.stream_callback,
            )
    
            self.worker_thread = threading.Thread(
                target=self.worker,
                daemon=True,
            )
    
            self.stop_signal = threading.Event()
            self.audio_queue = queue.Queue(maxsize=100)
            self.result_queue = queue.Queue()
    
        def __enter__(self):
            logger.info("Starting transcriber")
            self.input_stream.start()
            self.worker_thread.start()
            return self.gen()
    
        def gen(self):
            try:
                yield from self.get_result()
            except KeyboardInterrupt:
                logger.info("Keyboard interrupt received, shutting down...")
                pass  # Exit the iterator, GOTO __exit__
    
        def get_result(self):
            while not self.stop_signal.is_set():
                try:
                    text = self.result_queue.get(timeout=0.1)
                except queue.Empty:
                    continue
                yield text
    
        def stream_callback(self, data, _frames, _time, status):
            if status:
                logger.info("Audio status: {status}")
            try:
                self.audio_queue.put_nowait(data)
            except queue.Full:
                pass  # Drop audio if overflowing
    
        def worker(self):
            while not self.stop_signal.is_set():
                try:
                    data = self.audio_queue.get(timeout=0.1)
                except queue.Empty:
                    continue
    
                logger.debug("Got data")
                utterance = self.vad_buffer.process(data)
                if utterance is None:
                    continue
    
                logger.info("Got utterance")
                if self.callback is None:
                    self.result_queue.put(utterance)
    
                result = self.callback(utterance)
                if result is not None:
                    logger.info("Got result")
                    self.result_queue.put(result)
    
        def __exit__(self, *_):
            logger.info("Ending transcriber")
            self.input_stream.stop()
            self.stop_signal.set()
    
    
    def main():
        results = []
        with VoiceStreamer(record_device=0) as streamer:
            for result in streamer:
                results.append(result)
    
        return results
    
    
    if __name__ == "__main__":
        main()
  • Returning to R

    This week, I had a wonderful experience when returning to R. After several years of writing Python as a Machine Learning Engineer (and occasionally dabbling in Julia), The Tidyverse ecosystem in R astounded me with how short and expressive the functional style of programming was for analysing and visualising data.

    I first used R in an undergraduate statistics course (which was not a subject I enjoyed at the time). In the 11 years that passed, I went through mathematical neuroscience and ended up in the field of machine learning and data science — where Python reigns supreme, especially for neural networks.

    I was curious about returning to R to see how my perspective has changed, and I picked up a copy of “R for Data Science”. Before I knew it, I had fallen in love with the Tidyverse — and I absolutely understand why many data scientists swear by it.

    One of the best mantras I’ve come across for good software design comes from the title of a classic book on web design and usability by Steven Krug:

    Don’t make me think

    Great software is pleasant to use, because it gets out of your way and works intuitively. The principles that the Tidyverse is built on perfectly aligns with this philosophy

    • Design for humans
    • Reuse existing data structures
    • Design for the pipe and functional programming

    The end result: an ecosystem of packages that gives non-programmers a powerful system to handle any data science problem using short and clear code. Adopting a functional style from the get-go made code significantly shorter and easer to compose, debug, and understand.

    I experienced this first hand with the latest dataset from Tidy Tuesday

    library(tidyverse)
    library(tidytuesdayR)
    
    # Get data
    tuesdata <- tidytuesdayR::tt_load('2026-02-10')
    df <- tuesdata$schedule

    First I wanted to explore the schema of the dataset, here’s a 2 step pipeline I used:

    get_schema <- function(df) {
        tibble(
            col = df |> schema() |> names(),
            dtype = map_chr(df, ~ class(.x)[1]),
        )
    }
    
    summarise_schema <- function(df_schema) {
        df_schema |>
            group_by(dtype) |>
            summarise(count = n(), cols = paste(col, collapse = ", ")) |>
            arrange(desc(count))
    }
    
    df_raw |> get_schema() |> summarise_schema()

    Very little cleaning was needed for this dataset. Here’s how I coded a couple of the plots I used to investigate some of the posed questions:

    # How are medal events distributed across the days of the Olympics?
    plot_bar_medal_events_by_date <- function(df) {
        df |>
            group_by(date=as.Date(start_datetime_utc), discipline_name) |>
            ggplot(aes(x=date, fill=discipline_name)) +
            geom_bar()
    }
    
    df |> plot_bar_medal_events_by_date()
    
    # What is the typical duration of different types of events?
    plot_boxplot_duration_by_discipline <- function(df) {
        df |>
            mutate(duration = end_datetime_utc - start_datetime_utc) |>
            ggplot(aes(x=duration, y=discipline_name)) +
            geom_boxplot()
    }
    
    df |> plot_boxplot_duration_by_discipline()

    Comparing this with how I would write this in Python with pandas, the verbosity of Python would require using significantly more lines of code to express the same outcome. The resulting Python code would be more of a chore to write and harder to read — two aspects that run counter to a good exploratory data analysis task.

    Writing code — whether for software or data science — is an exercise in communication. Code that is easy to read for others (and your future self) is something we should all strive for. R and the Tidyverse is absolutely worth celebrating, because it makes this significantly easier for those without a background in computer science

    One fascinating blog post I read on this topic was written by Rasmus Bååth, who had the opposite experience as an R programmer entering the world of Python. I highly recommend reading their post for more concrete discussions and examples on R versus Python for data analysis.

  • Your AI Agent is Unsafe and Sandboxes Won't Save It

    It has been impossible to avoid the sheer amount of coverage of OpenClaw (formerly MoltBot, formerly ClawdBot). For those who have managed to avoid it, OpenClaw is an open source implementation of an AI agent that “can manage your life”. You can hook it up to a messaging app (such as Telegram or WhatsApp), install plugins to give access to your email and your calendars and hook it all up to an LLM backend such as ChatGPT, Claude, or Gemini.

    Essentially, it’s an AI secretary. OpenClaw has drastically simplified the process of integrating a CRON job and an LLM with large aspects of your digital life. While people are installing this in a “sandboxed” VPS and letting loose with this program, every part of my being shudders at the complete disregard for security. Adding insult to injury, this post on XDA claims to have found over 21,000 unsecured instances of OpenClaw!

    Did the last 40+ years never happen in terms of software security? Have we seriously learned nothing? Or are we so willing to throw these valuable lessons out the window as soon as a shiny toy comes along

    One of my favourite presentations on this topic is “the lethal trifecta” by Simon Willison on LLM security. I highly recommend checking it out, but one of the key takeaways from the talk is that an LLM that has

    1. Access to private data
    2. Ability to externally communicate
    3. Exposure to untrusted content

    Then you’ve stumbled into the lethal trifecta, and a major security breach is one prompt injection attack away

    In systems that interface with databases, tremendous care is taken to sanitise any user input so that a malicious actor can’t execute a query on a database by feeding in an SQL scripts into the user input. Today with LLMs, this type of attack is no longer bound to simple SQL statements — LLM prompt injections are written in plain natural language, which can easily be obfuscated within a larger body of text or hidden in a webpage via text that doesn’t render (such as a comment in HTML or SVG, or even plan text which is styled to render invisibly on a web browser)

    For the love of all that is holy and otherwise — DO NOT DO THIS

    Filtering out prompt injections is still an ongoing field of research, and it’s remarkably easy to circumvent any current measures put in place

    Using a sandbox for OpenClaw might be enough to prevent any destructive actions happening to the computer, but access to private data and ability to communicate with the web are fundamentally flawed modes of operation

    This is yet another LLM/AI software that is incredibly unsafe to use for people (especially those with a little expertise, at the top of the Dunning-Krueger curve). Another recent example is Google Antigravity, which again has almost unrestricted input from arbitrary webpages it processes, providing one of the largest attack vectors I have ever seen

    And what is the response or protection? A small disclaimer that people are already accustomed to tuning out

    While it is an exciting time, I’m incredibly disappointed by the negligent practices of AI developers churning out software to chase the hype at the expense of safety

    I take issue with a lot of the culture surrounding AI and LLMs and how their outputs are trusted more than they ought to be. We are on track for a challenger-level disaster, adding these massive security holes on top of that larger issue is beyond horrifying to me

  • Declaring Python Scripts with `pyproject.toml`

    Something I’m constantly forgetting is how to declare scripts with pyproject.toml. This is a quick reference for me to come back to when I inevitably forget this again.

    Let’s work with basic project structure:

    .
    |-- pyproject.toml
    `-- foo
        |-- __init__.py
        |-- __main__.py
        `-- ...

    The file __main__.py will be the function we want to run in the script:

    def main():
        print("Hello from my app!")
    
    
    if __name__ == "__main__":
        main()

    Define the entry point under [project.scripts] in pyproject.toml:

    [project]
    name = "my-app"
    version = "0.1.0"
    
    [project.scripts]
    foo = "foo.__main__:main" # Runs main() from foo/__main__.py
    
    [build-system]
    requires = ["hatchling"]
    build-backend = "hatchling.build"
    
    [tool.hatch.build.targets.wheel]
    packages = ["foo"]

    Now this can be executed with uv run foo if you’re using uv (which you should be in this age!), or using pip install -e . then calling foo