I’m Blake. I write about probability, stochastic processes, Python tooling, and the practical side of building things with code. More about me →

  • Quantifying Trust: Ensuring Reliability in LLM Summaries

    Building text summarisers with LLMs is a fascinating project to take on. Compressing complex documents is effortless, but establishing and maintaining trust in a system is still vital — especially in production settings. How do we know if we’ve picked the best model for the task? Can we establish a baseline before we attempt to fine-tune a model?

    I was unfamiliar with standard metrics to use for text data. I learned that classical metrics used in this field often fell apart when used for evaluating the “factuality” of long document summaries. Modern techniques revolve around embeddings, which embeddings provide a way to quantify semantic and conceptual meaning of text data (via cosine similarity). But still, two sentences can contradict each other but still have a high cosine similarity. This problem is even more pronounced as the size of the text increases and the embeddings lose their granular meaning.

    Natural language interface models (NLIs) provide a way to evaluate factuality, but they still fall apart when naively applied to long inputs.

    The publication on “LongDocFact Score” stuck out to me. Instead of naively comparing entailment scores on across every text and summary sentence pair, they extract the top K semantically similar sentences from the source for each sentence in the summary, add a bit of context (sentence before and sentence after) before computing the NIL scores on each snippet/summary sentence pair.

    We can illustrate this in Python using nltk and transformers:

    # Define texts and summary
    text = "..."
    summary = "..."
    
    
    # Segment text and summary
    import nltk
    
    nltk.download("punkt")
    
    def split_sentences_nltk(data: str) -> list[str]:
        return [s.strip() for s in nltk.sent_tokenize(data) if s.strip()]
    
    segments_text = split_sentences_nltk(text)
    segments_summary = split_sentences_nltk(summary)
    
    
    # Embed segments
    from sentence_transformers import SentenceTransformer
    
    device = "cuda"
    sentence_model_name = "all-MiniLM-L6-v2"
    sentence_model = SentenceTransformer(sentence_model_name)
    sentence_model.eval()
    sentence_model.to(self.device)
    
    def get_similarity_matrix(
        segments_text: list[str],
        segments_summary: list[str],
        sentence_model: SentenceTransformer,
    ) -> torch.Tensor:
        """Returns similarity matrix with shape (len(segments_text),
        len(segments_summary)).
        """
        norm_embedded_segments_text = torch.nn.functional.normalize(
            sentence_model.encode(
                segments_text,
                convert_to_tensor=True,
                show_progress_bar=False,
            ),
            p=2,
            dim=1,
        )
        norm_embedded_segments_summary = torch.nn.functional.normalize(
            sentence_model.encode(
                segments_summary,
                convert_to_tensor=True,
                show_progress_bar=False,
            )
            p=2,
            dim=1,
        )
        return (
            norm_embedded_segments_text @ norm_embedded_segments_summary.T
        )
    
    similarity_matrix = get_similarity_matrix(
        segments_text,
        segments_summary,
        sentence_model,
    )

    Once we’ve constructed the similarity matrix, we can construct the snippets for each sentence in the summary:

    # Get top k similar text segments with context for each summary segment
    k = 3
    
    _, top_k_indices = torch.topk(similarity_matrix, k=k, dim=0, largest=True)
    
    
    def get_snippet(data_sen: list[str], doc_idx: int) -> str:
        context_parts = []
        if doc_idx > 0:
            context_parts.append(data_sen[doc_idx - 1])
    
        context_parts.append(data_sen[doc_idx])
    
        if doc_idx < len(data_sen) - 1:
            context_parts.append(data_sen[doc_idx + 1])
    
        return " ".join(context_parts)
    
    
    top_k_context_segments = [
        [get_snippet(data_sentences, k) for k in top_k_indices[:, idx]]
        for idx in range(similarity_matrix.shape[1])
    ]

    Now, we have a much more efficient way to calculate the scores for this pair of documents:

    # Calculate scores for each summary segment
    from typing import Any
    import transformers
    import torch
    
    model_name = "cross-encoder/nli-deberta-v3-base"
    tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
    model = transformers.AutoModelForSequenceClassification.from_pretrained(model_name)
    self.model.to(self.device)
    
    label_idxs = {
        "contradiction": 0,
        "entailment": 1,
        "neutral": 2,
    }
    
    
    def flatten_list_of_lists(list_of_lists: list[list[Any]]) -> list[Any]:
        return [value for list_of_values in list_of_lists for value in list_of_values]
    
    
    def explode_list_by_k(list_of_values: list[Any], k: int) -> list[Any]:
        return [v for v in list_of_values for _ in range(k)]
    
    
    features = {
        k: v.to(self.device)
        for k, v in tokenizer(
            flatten_list_of_lists(top_k_context_segments),
            explode_list_by_k(segments_summary, k),
            padding=True,
            truncation=True,
            return_tensors="pt",
        ).items()
    }
    
    with torch.no_grad():
        logits = self.model(**features).logits
        scores = softmax(logits, dim=-1)
    
    n_snippets, n_labels = scores.shape
    scores, _ = scores.reshape(int(n_snippets / k), k, n_labels).max(dim=1)
    
    metrics = {
        "mean_entailment": float(scores[:, label_idxs["entailment"]].mean().item()),
        "mean_contradiction": float(scores[:, label_idxs["contradiction"]].mean().item()),
        "max_contradiction": float(scores[:, label_idxs["contradiction"]].max().item()),
    }
  • The Balance that Builds

    Progress demands consistency, but consistency doesn’t survive on willpower alone. It needs the balance of joy and discipline. Reflecting on my post last week, I thought there was more to say about how I’ve grappled with this balance and where I’ve landed after mistakes and lessons over the years.

    I don’t write much about weightlifting, despite doing it for over six years now. But there are some key parallels with the barbell that helped me understand this balance, and more importantly, allowed me to practice it. That practice carried over into other parts of my life.


    If you don’t like it, you wont do it

    Discipline matters, but joy is the real glue that holds consistency together. I’ve seen this play out countless times in the gym: if you dread every session, you’ll eventually find reasons not to show up. The same goes for work, study, or any long-term project. If you can’t find a way to enjoy the process — even in small ways — it becomes unsustainable.

    For me, this meant experimenting until I found a training style I actually looked forward to, not just one I thought I should be doing. That shift made it possible to keep showing up, week after week, year after year.


    You can only perform as well as you rest

    Take your rest seriously: put yourself and your well-being before anything else. While challenge and pressure are necessary for growth, too much of either leads to burnout or injury. The physical effort in lifting is the stimulus for progress — not the progress itself! More than 80% of real progress happens during recovery — getting good quality sleep and maintaining good nutrition so that you have the fuel you need to recover and adapt.

    I reached a point where I felt like I was plateauing. I took my rest seriously and I put in the full effort every week, but I could not make any progress. Counter-intuitively, the solution was to pull back and “deload” every 4 weeks. This didn’t mean stepping away entirely (though that is needed from time to time). But I was finally able to keep progressing once I regularly started to take a week to work at 60% effort and focus more on the foundations (such as mobility, flexibility and balance).


    Bumps in the road do not matter in the long term

    A bad week does not make a bad month. Setbacks are inevitable, but shame and perfectionism make them heavier than they need to be. What matters is momentum. As long as you keep moving, the missed days, the missteps, even the failures fade into the background.

    This has helped me let go of the illusion of perfect progress. A week where nothing goes to plan doesn’t erase the months of steady work that came before it. In the long run, consistency beats perfection every time.


    So take care of yourself and keep your eyes on the long game. Making better choices pays off

  • Excellence is a byproduct of having fun

    Excellence in anything is a byproduct of having fun. Fun is a byproduct of understanding. Understanding is a byproduct of going slow. Going slow is a byproduct of curiosity. Curiosity is a byproduct of saying “I don’t know,” of shunning beliefs and attending to what is in front, with zero baggage or impositions of your own — shunning the ego in the moment, moment by moment.

    Excellence comes when each piece is as equal as any other, when preference is shunned, when space is created to allow what is in the moment, without resistance, without insistence.

    This comment by jimsojim resonated with me when I read it this week. Beyond programming, I’ve experienced similar reflections and revelations across my other hobbies and other parts of my life.

    Digging deeper, I think the reason this resonates with me comes down to my personal philosophy and what drives me. If I could summarise my objective, it would be along these lines:

    I want to build cool shit with cool people, I want to share knowledge with those around me, and I want to use my skills and understanding to help others.

    Coupled with this, one of the greatest joys I know is the arc from not knowing to knowing — the slow, shared process of discovery. There’s a particular satisfaction in starting with nothing but questions, working alongside others, and watching our collective understanding take shape.

    I think those consistent dopamine hits are responsible for some incredible achievements. It’s why I believe in an iterative process — shunning perfectionism and ego (which is much easier said that done) to allow yourself to be fail and be humble. Then, finding your footing in a small win, and gaining momentum and confidence to drive towards bigger wins

    The hardest step is slowing down. Slowing down feels like falling behind. But it’s hard to maintain speed without momentum.

    I’ve learned that if I want “excellence”, I need to stop chasing it. Start by saying “I don’t know”, and learn to listen more. Pay attention to what’s in front of you without trying to control it. Make space for curiosity. Fun will follow — and excellence will take care of itself.

  • TIL: Fixing an incorrect Git rebase

    Have you ever found yourself in this situation? You’ve used git rebase to clean up commit history or integrate changes from another branch. And now your changes appear out of sync, and somehow Git thinks you’ve deleted code from the branch you initially started from.

    This happens because rebasing fundamentally alters Git history rather than preserving it.

    Each commit is a snapshot of the repo at a particular point in time, and the order of these snapshots matters immensely. When you rebase, you’re essentially taking your commits and replaying them on top of a different base commit, which changes their context and can create conflicts with the existing history.

    Unfortunately, this fundamental concept isn’t made clear when using Git in day-to-day operations — the commands feel simple, but they’re performing complex history manipulation under the hood.

    Prevention is Better Than Cure

    In my opinion, the way to avoid this is to use git merge instead of git rebase unless you absolutely need to rewrite history. Here’s why:

    • Rebases change history - they rewrite commits, which can cause confusion when collaborating
    • Merges add to history - they preserve the original timeline of your work
    • Merges record context - they capture exactly what changes happened during the merge process

    Once you’ve rebased incorrectly, a simple merge after the fact won’t solve the problem. The changes you made happened after those latest commits, so Git still believes you’ve deleted legitimate code.

    The Solution: Diff and Apply

    We’re trying to pull changes from <other-branch> into <my-branch>. After a bad rebase, the changes from <other-branch> are nowhere to be found. We’ll head to the terminal to fix it:

    Let’s list the files that are different:

    git diff HEAD..<other-branch> --name-only

    The HEAD..<other-branch> syntax means “show me what changes will take me from HEAD (my current commit) to <other-branch>” — essentially, what’s in the other branch that I’m missing.

    We only want to merge changes from a subset of files, so let’s exclude the files we don’t want to patch. (For example, let’s exclude the file uv.lock and all markdown files)

    git diff HEAD..<other-branch> --name-only -- ':(exclude)uv.lock' ':(exclude)**/*.md'

    Once we’re happy with the list of files, let’s make a patch:

    git diff HEAD..<other-branch> -- ':(exclude)uv.lock' ':(exclude)**/*.md' > patch.diff

    Open the file patch.diff in your editor — we can see all the changes that would occur if we apply the patch. There might be some unrelated changes, so let’s edit the patch and remove the changes we don’t want and save it.

    Now we can apply the changes:

    git apply patch.diff

    There may be some conflicts, edit the files and resolve the conflicts one by one. Once we’re finished, we can remove the patch.diff file.

    Now git add your changes and git commit as you normally would.

    All good now!