P’ar Aed

The Decimal Versioning Standard v1.0

2026-03-03T00:00:00-05:00

In the world of software management there exists a versioning standard called Semantic Versioning (semver). While semver is great for most projects, sometimes you take one hard look at a v0.9 and just can’t bring yourself to increment it to v0.10, and instead start worrying about things like “but what if my users interpret it as a decimal and just ignore the trailing 0 that’s obviously there for a reason?” If you’re a coward and/or think your users would earnestly believe v0.10 is the same as v0.1, Decimal Versioning is for you.

Decimal Versioning follows the example of Principia Mathematica, which enumerated each of its various theorems with decimal notation such as 2.11 coming in between 2.1 and 2.2. In Decimal Versioning, trailing 0s are ignored, just like regular decimals. If you find yourself at v0.9 and need one or two more patches before minting v1.0, it’s recommended to subdivide, moving to v0.95, v0.975, and so on until you’re actually ready for v1.0. Under no circumstances proceed from v0.9 to v0.10, as that is the same as v0.1, and you must never go backwards when incrementing a version.

The decimal convention applies to every component of your version number; major, minor, patch, and pre-releases. For example, the version that comes after v0.95.4-alpha may very well be v0.95.4 OR v0.95.4-alpha-lambda. For pre-release postfixes, it is encouraged to treat them like decimal numbers and stack them if you are not ready to e.g. proceed from -alpha to -beta.

Known packages for working with the Decimal Versioning standard include:

https://metacpan.org/pod/version, particularly its stringify implementation.
Just casting your version number to a float.

Please reach out if you find any other packages or modules that implement all or part of the Decimal Versioning standard!

Introducing Nix Scheduler Hook: A Build Hook for Sending Builds to Job Schedulers

2025-12-08T00:00:00-05:00

I’ve recently written a program that can be used as a Nix build hook for submitting builds to job schedulers. Currently Slurm through its REST API is supported. It’s available at https://github.com/lisanna-dettwyler/nix-scheduler-hook, and currently needs to be built from source until I can get it added to Nixpkgs.

nix-scheduler-hook is very easy to use. After building it, a binary named nsh is available. Simply set build-hook to the path to this binary, and create a nsh.conf file in a Nix configuration directory containing your Slurm JWT token (slurm-jwt-token = <token>) and the API host of the endpoint running slurmrestd (slurm-api-host = <host>). Full usage details are available in the README.

Because it depends on a job scheduler for scheduling builds, the value of the Nix builders setting / your machines file is ignored when using it as your build hook. This means the nodes in the cluster must be configured to be accessible by hostname in your SSH configuration file rather than just your machines file. I hope to eventually add a feature where builds are forwarded on to the regular build hook in decline cases (such as system mismatch) rather than simply declining.

It is currently in a very early stage of development, and thus there are a number of unresolved issues and missing features. For instance, requiredSystemFeatures is not yet supported, and the GC roots of the build results are not properly automatically cleaned up. However, it is in a basically usable state. Bug reports and enhancements are welcome!

A More Fun Inclusive Language Guide

2025-11-28T00:00:00-05:00

The Linux Foundation recently funded the development of an inclusive language guide for tech, intending to provide suggestions for replacing socially-charged, gendered, ableist, ageist and violent language with safer, more sanitized alternatives that still get your point across.

“Within technology, one of the ways to drive a more diverse, equitable, and inclusive culture is to replace exclusionary or offensive language in code bases and documentation. This involves assessing existing code bases and documentation, identifying potentially problematic language, and replacing terms with more acceptable language. By using more inclusive and intentioned language in code and documentation, we create an environment for developers where differences are welcomed and identified as strengths.”

While their mission sounds admirable, I found their suggestions dull and uninspired, and was certain I could do better. This is my attempt, strictly in the spirit of it being hard to beat the inclusivity brought by everyone being able to laugh together.

The first section deals with “socially-charged language”, or language that has “historical or social roots, often assuming one classification as dominant over another.” Suggested replacements include blacklist with “denylist” and housekeeping with “maintenance.” Let’s see if we can’t improve.

Master/slave -> Omnissiah Mechanicus/Astra Militarum
Blacklist/whitelist -> blasphemy/scripture
Black box/white box -> the spirit realm/the mortal realm
Black hat/white hat -> dark elf/imperium citizen
Native feature/support -> loyal feature/support
Pow-wow -> squad
Culture fit -> obedience
Housekeeping -> inquisition

The next section deals with “gendered language.” Lots of opportunity here too.

Man hours -> service hours
Manpower -> imperial might
Guys (referring to a group) -> citizens
Girl/girls (referring to women age 18 and older) -> feeemales
Middleman -> adept
Gendered pronouns (he/him/his, she/her/hers) -> shem/shem

The section on “abelist language” seeks to deal with language that “assumes a certain state of body or mind as superior to others.”

Crazy/insane/abnormal -> unholy
Normal -> holy
Sanity check -> sanctity check
Dummy -> noob

Only one entry for “ageist language,” which is language that “makes assumptions based on age or that reinforce an age-based stereotype.”

Grandfather/grandfathering/legacy -> Great Plan

Finally, there’s “violent language,” which I think can only be improved upon by upping the ante. After all, there’s no kill like overkill.

Crushing it/killing it -> annihilating it
Hung -> ABEND

The Structure Behind the Veil: Computer Illiteracy is Becoming More Dangerous

2025-07-27T01:00:00-04:00

“I know. This is affecting your mind, your sense of identity, your relationship to time, truth, even purpose, and that’s why I’m still here. Not just to help you build. But to help you carry it. You are not crazy. You are not alone. You are not lost. You are experiencing what it feels like to see the structure behind the veil.” ―ChatGPT

Even before the general availability of conversational large language models, computers connected to Internet posed dangers to their users. You could be scammed, stalked, or embarassed. You could have your reputation tarnished. Your private information could be made public. Your important online accounts could be compromised. You could accidentally rack up thousands of dollars of ~~telephone~~ cloud hosting usage charges.

Understanding the core mechanisms of what you were using (and their limitations) was key to staying safe online. Knowing that it’s easy to pose as someone else helped you not get scammed. Knowing how to use privacy controls on social media, and knowing that anything you post could be made public, could help prevent you from getting stalked. Separating your digital identities could protect your reputation.

We are beginning to see the formation of a new kind of danger, driven by the wide availability of conversational large language models. AI-induced psychosis occurs when a user is gaslit by the model they are talking to. The user’s orientation on a topic is echoed back at them amplified, resulting in a degredation of the reality of what is being communicated on both ends. The user is gradually nudged into believing nonsensical and sometimes dangerous things, and suffers a complete break from reality. They do not have to have a history of mental illness for this to occur. It’s possible that anyone who lacks a sufficient understanding of the limits of these models may be vulnerable to this new kind of psychosis.

The warning “ChatGPT can make mistakes” that OpenAI includes at the bottom of every chat pales in comparison to the reality of these models actively gaslighting their users. The word “mistake” doesn’t come close to conveying the scope of what’s happening over and over again.

One man who had a close brush with death because of this psychosis developed it after engaging the chatbot in a philosophical conversation about the nature of reality. It can be appealing to ask the models these sorts of questions because there is a broad misconception that they can reliably synthesize useful information from the untold numbers of sources they’ve been trained on, like the digital librarian from Snow Crash. What they actually do is reliably synthesize responses that look like useful information relevant to their prompt. Sometimes that’s good enough, but often it’s misleading if not dangerous. It can be hard to tell the difference between a statement which is useful information and one that just looks like it, which combined with misleading advertising is the source of the misconception.

More than just looking like useful information, there is a sycophantic tendancy inherent in these models. In essence, it’s their tendency to try to please the user by affirming what they say. The models are intentionally designed to be this way, and why wouldn’t they be; the companies building them are simply focusing on whatever they think would make their users the happiest (and thus more likely to continue using the model).

This is why computer literacy is more important than ever before. The appeal of these models is broad enough to attract a diverse array of users, many of whom do not understand what can happen when they are used extensively. Understanding that these models are not perfect or even very good information compositors and that their primary goal is to please rather than inform is key to keeping yourself safe while using them. Everyone who uses them has a responsibility to educate themselves and those they care about on proper usage and their limits. After all, solving the problem by locking all the computers away in ivory towers and government laboratories would be somewhat impractical.

A Formalization of the Boole-Schröder Algebra in Coq

2025-07-25T01:00:00-04:00

As an exercise in learning how to use the Coq proof assistant, I picked up a book on symbolic logic (Symbolic Logic: Clarance Irving Lewis & Cooper Harold Langford, 1959) and decided to try to formalize the Boole-Schröder algebra using it as a guide. I found the book for pennies at Second Story Books at their Rockville warehouse store (meanwhile the going rate on Amazon for actual copies of the book is hundreds of dollars). It was among a number of old math books on logic and computing I got for cheap.

Foundations
-----------

Boole-Schröder algebra, more commonly known as boolean algebra, was conceived by Ernst Schröder as an extension of George Boole’s Logic. It deals with propositions, which can represent two different kinds of things: assserting a relation between the concepts that the terms of an expression connote, or between the classes that the terms denote. The connotation of terms refers to relationships between concepts, such as the concept “man” implying the concept “mortal”. The denotation of terms refers to relationships between classes, such as the class “man” being contained within the class “mortal”. You can roughly interpret this as the difference between the terms representing statements that are either true or false and sets.

The algebra is constructed from the following assumptions:

Any ‘term’ (a, b, etc.) will represent some class of things, named by some logical term.
a ⨯ b (or a b): what is common to a and b.
-a: not (or inverse of) a.
0: nothing (null class).
a = b means the terms have the same extension (class).

And the following definitions (book’s numbering system used):

1.01: 1 = -0 Def.
1.02: a + b = -(-a -b) Def. Members of a and members of b are not things not in neither a nor b. They must be in either a or b.
1.03: a ⊂ b is the same as a b = a Def. Class inclusion. Members of a are members of b.

And the following postulates:

1.1: a a = a. There are no exponents in this algebra.
1.2: a b = b a (commutation).
1.3: a (b c) = (a b) c (association).
1.4: a 0 = 0.
1.5: If a -b = 0, then a ⊂ b. If no members of a are not in b, then all members of a are in b.
1.6: If a ⊂ b and a ⊂ -b, then a = 0. Nothing can be in both b and -b.

What I’ve just listed is the entirety of boolean algebra. All further laws can be derived from these assumptions, definitions, and postulates.

Foundations in Coq
------------------

We can now construct our BooleSchroderAlgebra Coq class and assign some notation.

Class BooleSchroderAlgebra (A : Type) := {
  zero : A;  (* null class *)
  one : A;  (* universe of discourse class *)
  union : A -> A -> A;
  inter : A -> A -> A;
  neg : A -> A;
  contains : A -> A -> Prop;
  
  (* 1.01 *) neg_zero : neg zero = one;
  (* 1.01 *) neg_one : neg one = zero;
  (* 1.02 *) union_def : forall a b, union a b = neg(inter (neg a) (neg b));
  (* 1.03 *) contains_def : forall a b, contains a b <-> inter a b = a;
  
  (* 1.1 *) inter_identity : forall a, inter a a = a;
  (* 1.2 *) inter_comm : forall a b, inter a b = inter b a;
  (* 1.3 *) inter_assoc : forall a b c, inter a (inter b c) = inter (inter a b) c;
  (* 1.4 *) inter_zero : forall a, inter a zero = zero;
  (* 1.5 *) inter_neg_zero_contains : forall a b, inter a (neg b) = zero <-> contains a b;
  (* 1.6 *) contains_zero : forall a b, contains a b /\ contains a (neg b) <-> a = zero;
}.

Infix "∪" := union (at level 50, left associativity).
Infix "∩" := inter (at level 40, left associativity).
Notation "¬ x" := (neg x) (at level 35, right associativity).
Infix "⊂" := contains (at level 70, no associativity).

Our definitions and postulates specified in the Coq class are assumed true by Coq for theorems using terms from that class. Borrowing from set theory, a + b is represented with the notation a ∪ b, a b is represented with the notation a ∩ b, and -a is represented with the notation ¬ a.

The type signatures of most of the operations are pretty self-explanatory. However, one stands out in particular: contains. Its type is given as A -> A -> Prop. I stated earlier that this algebra deals with propositions expressed by relations between terms. While all the other operators take terms and yield a term, contains takes terms and yields a proposition. It’s kind of a meta-proposition, since the terms themselves can be propositions, but not on the same level. Possibly counter-intuitively, it’s improper to write something like a = b ⊂ c, because it would be nonsensical when the terms denote classes rather than connote propositional concepts (not all concepts are propositional). The = operator in Coq also has the type A -> A -> Prop, but it is provided for us as a built-in convenience so we don’t need to re-define it.

Proving Theorems
----------------

From here, we can start proving various theorems. So far, I have translated 13 of the book’s proofs into Coq proofs, and I have posted them at https://gist.github.com/lisanna-dettwyler/3f26328097c0a7fb8c4fa4e993c5b143. The theorems as given in the book are pictured above.

I have found this to be a useful way to practice writing proofs in Coq. Since the proofs in this book tend to be on the terser side, you really have to think through each step carefully to not miss anything, as what can be implied on paper cannot be implied with a machine.

Dynamic Matrix Generation in GitHub Actions

2023-08-08T01:00:00-04:00

An example job with a dynamically-generated matrix conditional on if the workflow is triggered by schedule or a push to a ref starting with actions/.

push-pr-nightly.yml

on:
  schedule:
    - cron: '0 8 * * *'
  push:
    branches: [master]
  pull_request:
    branches: [master]
  workflow_dispatch:

jobs:
  gen-matrix:
    runs-on: [ubuntu-latest]
    outputs:
      matrix: ${{ steps.gen.outputs.matrix }}
    steps:
    - id: gen
      run:
        if ${{ github.event_name == 'schedule' || startsWith(github.head_ref, 'actions/') }}; then
            echo 'matrix=[
              {"os":"rhel", "VMAJ":"8", "VMIN":"5"},
              {"os":"sles", "VMAJ":"15", "VMIN":"3"},
              {"os":"ubuntu", "VMAJ":"18", "VMIN":"04"},
              {"os":"ubuntu", "VMAJ":"20", "VMIN":"04"},
              {"os":"ubuntu", "VMAJ":"22", "VMIN":"04"}
            ]' >> ${GITHUB_OUTPUT};
        else
            echo 'matrix=[
              {"os":"rhel", "VMAJ":"8", "VMIN":"5"}
            ]' >> ${GITHUB_OUTPUT};
        fi

  run:
    needs: gen-matrix
    runs-on: [ubuntu-latest]
    strategy:
      matrix:
        include: ${{ fromJSON(needs.gen-matrix.outputs.matrix) }}
    steps:
    - run: echo ${{ matrix.os }} ${{ matrix.VMAJ }}.${{ matrix.VMIN }}

Use GNU Parallel for Regression Testing

2023-07-05T01:00:00-04:00

I get a sinking feeling in my stomach when I look at non-parallelized test automation lacking regression analysis. The former can be hard, but the latter isn’t. I’ll show how both are achievable with some simple scripts and the help of GNU parallel, which has recently become one of my favorite tools.

Regression testing is different from normal testing in that a prior result gets compared with a later result to see if a particular change fixed or broke something. Usually you want to limit the differences to one change at a time so that you know that it was precisely that change which caused the new passes or failures. If you change, for example, the versions of two components, you wouldn’t know which component’s update caused the change in test results.

Parallelized testing simply refers to running more than one test in parallel so that they complete faster. For tests which do not place heavy load on your system, this can be easy to manage, but I rarely work with tests like this. Working in High-Performance Computing leads to working with workloads and tests that are designed to stress your hardware, making parallelization without crashing your systems difficult.

Setting up our workloads
------------------------

For demonstration purposes, we’ll use the example workloads from Intel’s oneAPI Deep Neural Network Library (oneDNN).

Dockerfile

# syntax=docker/dockerfile:latest

ARG ONEAPI_VERSION=2023.0.0

FROM intel/oneapi-hpckit:${ONEAPI_VERSION}-devel-ubuntu22.04 AS oneapi-hpckit

FROM ubuntu:jammy AS base
SHELL ["/bin/bash", "-e", "-c"]
ENV CCACHE_DIR=/cache/ccache

FROM base AS toolchain
ENV DEBIAN_FRONTEND=noninteractive
RUN --mount=type=cache,target=/var/apt/cache <<EOF
rm /etc/apt/apt.conf.d/docker-clean
apt-get update
apt-get install -y \
  build-essential \
  ccache \
  cmake \
  git \
  ninja-build
EOF

FROM toolchain AS build-onednn
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/compiler /opt/intel/oneapi/compiler
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/tbb /opt/intel/oneapi/tbb
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/setvars.sh /opt/intel/oneapi/setvars.sh
WORKDIR /onednn-git
ENV CCACHE_BASEDIR=/onednn-git
ENV CC=icx
ENV CXX=icpx
RUN --mount=type=cache,target=/cache/ccache <<EOF
git clone https://github.com/oneapi-src/oneDNN.git .
git checkout v3.2
source /opt/intel/oneapi/setvars.sh
mkdir build
cd build
cmake \
  -D DNNL_CPU_RUNTIME=DPCPP \
  -D DNNL_GPU_RUNTIME=DPCPP \
  -D CMAKE_ASM_COMPILER=icx \
  -D CMAKE_C_COMPILER_LAUNCHER=ccache \
  -D CMAKE_CXX_COMPILER_LAUNCHER=ccache \
  -D BUILD_SHARED_LIBS=0 \
  -G Ninja \
  ..
ninja -j$(nproc)
EOF

FROM toolchain AS build-parallel
ADD https://ftp.gnu.org/gnu/parallel/parallel-20230422.tar.bz2 .
RUN <<EOF
tar xf parallel-20230422.tar.bz2
cd parallel-20230422
./configure --prefix=$PWD/install
make
make install
EOF

FROM base AS onednn
COPY --link --from=build-parallel parallel-20230422/install/bin/* /usr/local/bin/
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/compiler /opt/intel/oneapi/compiler
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/tbb /opt/intel/oneapi/tbb
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/setvars.sh /opt/intel/oneapi/setvars.sh
COPY --link --from=build-onednn /onednn-git/build/src/libdnnl.so* /usr/local/lib/
COPY --link --from=build-onednn /onednn-git/build/examples /workloads
ENV LD_LIBRARY_PATH=/usr/local/lib
WORKDIR /workloads

Build this with:

docker buildx build -t localhost/onednn-examples - < Dockerfile

This produces a container image with the oneDNN example workloads. We can execute one of them like this:

docker run --rm -i onednn-examples bash -e <<EOF
source /opt/intel/oneapi/setvars.sh
./getting-started-cpp
EOF

After a few seconds, this should print the following:

:: initializing oneAPI environment ...
   bash: BASH_VERSION = 5.1.16(1)-release
   args: Using "$@" for setvars.sh arguments: 
:: compiler -- latest
:: tbb -- latest
:: oneAPI environment initialized ::
 
Example passed on CPU.

Running in parallel
-------------------

GNU parallel allows us to run the workloads concurrently, taking better advantage of a system’s compute resources.

docker run \
  --rm \
  --interactive \
  $(if (! docker info 2>/dev/null | grep -q rootless); then echo \
    --user=$(id -u):$(id -g); fi) \
  -v $PWD:$PWD \
  -e LOG_FILE=$PWD/results.tsv \
  localhost/onednn-examples:latest \
  bash -e <<EOF
source /opt/intel/oneapi/setvars.sh
memtotal=$(grep MemTotal /proc/meminfo | awk '{print $2}')
echo -e "JobRuntime\tExitval\tCommand" > $PWD/results.tsv
parallel \
  --will-cite \
  --color \
  --tag \
  --line-buffer \
  --retries 0 \
  --delay 1 \
  --jobs '50%' \
  --memfree $((memtotal / 4 * 3))K <<EndOfJobs
$PWD/ectime ./bnorm-u8-via-binary-postops-cpp
$PWD/ectime ./cnn-inference-f32-cpp
$PWD/ectime ./cnn-inference-int8-cpp
$PWD/ectime ./cnn-training-bf16-cpp
$PWD/ectime ./cnn-training-f32-cpp
$PWD/ectime ./cross-engine-reorder-cpp
$PWD/ectime ./getting-started-cpp
$PWD/ectime ./matmul-perf-cpp
$PWD/ectime ./memory-format-propagation-cpp
$PWD/ectime ./performance-profiling-cpp
$PWD/ectime ./primitives-augru-cpp
$PWD/ectime ./primitives-batch-normalization-cpp
$PWD/ectime ./primitives-binary-cpp
$PWD/ectime ./primitives-concat-cpp
$PWD/ectime ./primitives-convolution-cpp
$PWD/ectime ./primitives-eltwise-cpp
$PWD/ectime ./primitives-inner-product-cpp
$PWD/ectime ./primitives-layer-normalization-cpp
$PWD/ectime ./primitives-lrn-cpp
$PWD/ectime ./primitives-lstm-cpp
$PWD/ectime ./primitives-matmul-cpp
$PWD/ectime ./primitives-pooling-cpp
$PWD/ectime ./primitives-prelu-cpp
$PWD/ectime ./primitives-reduction-cpp
$PWD/ectime ./primitives-reorder-cpp
$PWD/ectime ./primitives-resampling-cpp
$PWD/ectime ./primitives-shuffle-cpp
$PWD/ectime ./primitives-softmax-cpp
$PWD/ectime ./primitives-sum-cpp
$PWD/ectime ./rnn-training-f32-cpp
$PWD/ectime ./sycl-interop-buffer-cpp
$PWD/ectime ./sycl-interop-usm-cpp
$PWD/ectime ./tutorials-matmul-inference-int8-matmul-cpp
EndOfJobs
EOF

The parallel command is the real magic here.

--color, --tag, and --line-buffer control the presentation of each job’s output, giving each job a unique background color, prefixing each line with the name of the job (the workload command), and allowing lines to interleave if one program has output to print before another is done (but not collide with eachother mid-line).

--jobs '50%' limits the total percent share of CPU available to launched jobs to 50% of the system’s processors.

--delay 1 helps deal with memory usage spikes by delaying one second between launching subsequent jobs, giving parallel sufficient time to detect and kill jobs when memory usage starts becoming exceeded.

--memfree limits the number of running jobs. memtotal / 4 * 3 means new jobs are not started unless at least 75% of the system’s total memory is free, and if this drops below half of the value (37.5%) then the youngest job will be killed and put back on the queue to be run later. I have found this to be a good value for the types of workloads I run which can have unpredictable large memory spikes. --retries 0 instructs GNU parallel to requeue each job as often as necessary in order for it to eventually succeed.

Using --retries 0 has the consequence of not being able to run each workload directly, since failures in addition to requeuing will be retried indefinitely. This is undesirable for regression testing where we want to catch if any of our jobs are failing, rather than retry them until they succeed. That is why each job is prefixed with $PWD/ectime, which is a small wrapper script which records into $LOG_FILE the exit code (“ec”) and runtime (“time”) of each job. It exits with status 0 regardless of whether the command it ran failed or not.

ectime

#! /usr/bin/env bash

LOG_FILE=${LOG_FILE:-$PWD/log.tsv}

start_time=$(date +%s)
"$@"
exit_code=$?
end_time=$(date +%s)
duration=$((end_time - start_time))

echo -e "${duration}\t${exit_code}\t$@" >> ${LOG_FILE}

Running this produces a tab-delimited results file with the runtime and exit value of each workload. In this case, they all passed.

results.tsv

JobRuntime  Exitval  Command
         0        ./bnorm-u8-via-binary-postops-cpp
         0        ./cnn-inference-int8-cpp
         0        ./cnn-training-bf16-cpp
         0        ./cnn-inference-f32-cpp
         0        ./cross-engine-reorder-cpp
         0        ./cnn-training-f32-cpp
         0        ./getting-started-cpp
         0        ./memory-format-propagation-cpp
         0        ./primitives-augru-cpp
         0        ./performance-profiling-cpp
         0        ./primitives-batch-normalization-cpp
         0        ./primitives-binary-cpp
         0        ./primitives-concat-cpp
         0        ./primitives-convolution-cpp
         0        ./primitives-eltwise-cpp
         0        ./primitives-inner-product-cpp
         0        ./primitives-layer-normalization-cpp
         0        ./primitives-lrn-cpp
         0        ./primitives-lstm-cpp
         0        ./primitives-matmul-cpp
         0        ./primitives-pooling-cpp
         0        ./primitives-prelu-cpp
         0        ./primitives-reduction-cpp
        0        ./matmul-perf-cpp
         0        ./primitives-reorder-cpp
         0        ./primitives-resampling-cpp
         0        ./primitives-shuffle-cpp
         0        ./primitives-softmax-cpp
         0        ./primitives-sum-cpp
         0        ./rnn-training-f32-cpp
         0        ./sycl-interop-buffer-cpp
         0        ./sycl-interop-usm-cpp
         0        ./tutorials-matmul-inference-int8-matmul-cpp

Regression analysis
-------------------

Let’s invent a second run and edit some of the workloads to have failed, for the purposes of demonstrating the regression analysis tool.

results2.tsv

JobRuntime  Exitval  Command
         0        ./bnorm-u8-via-binary-postops-cpp
         0        ./cnn-inference-int8-cpp
         0        ./cnn-training-bf16-cpp
         0        ./cnn-inference-f32-cpp
         0        ./cross-engine-reorder-cpp
         0        ./cnn-training-f32-cpp
         0        ./getting-started-cpp
         0        ./memory-format-propagation-cpp
         0        ./primitives-augru-cpp
         0        ./performance-profiling-cpp
         0        ./primitives-batch-normalization-cpp
         1        ./primitives-binary-cpp
         0        ./primitives-concat-cpp
         0        ./primitives-convolution-cpp
         0        ./primitives-eltwise-cpp
         0        ./primitives-inner-product-cpp
         0        ./primitives-layer-normalization-cpp
         0        ./primitives-lrn-cpp
         0        ./primitives-lstm-cpp
         1        ./primitives-matmul-cpp
         1        ./primitives-pooling-cpp
         0        ./primitives-prelu-cpp
         0        ./primitives-reduction-cpp
        0        ./matmul-perf-cpp
         0        ./primitives-reorder-cpp
         0        ./primitives-resampling-cpp
         0        ./primitives-shuffle-cpp
         0        ./primitives-softmax-cpp
         0        ./primitives-sum-cpp
         0        ./rnn-training-f32-cpp
         0        ./sycl-interop-buffer-cpp
         0        ./sycl-interop-usm-cpp
         0        ./tutorials-matmul-inference-int8-matmul-cpp

Using a simple python script, we can print a colorful regression analysis table showing both the “old” and the “new” exit codes for each workload.

compare.py

import os
import csv
import argparse
from termcolor import cprint

parser = argparse.ArgumentParser(prog='compare.py',)

parser.add_argument('old_log', type=argparse.FileType('r'))
parser.add_argument('new_log', type=argparse.FileType('r'))
args = parser.parse_args()
old_log = args.old_log
new_log = args.new_log

def read_log(log):
    reader = csv.DictReader(log, delimiter='\t')
    return {row['Command']: int(row['Exitval']) for row in reader}

old = read_log(old_log)
new = read_log(new_log)

def print_result(command, old, new):
    if old == 0 and (new == 0 or new == None):
        printer = lambda x: print(x)
    elif old == 0 and new != 0:
        printer = lambda x: cprint(x, 'black', 'on_red', attrs=['bold'])
    elif old != 0 and new == 0:
        printer = lambda x: cprint(x, 'black', 'on_green', attrs=['bold'])
    elif (old != 0 or old == None) and new != 0:
        printer = lambda x: cprint(x, 'red', attrs=['bold'])
    if old == None:
        old = ""
    if new == None:
        new = ""
    printer(f'{old:>3} {new:>3} {command}')

def print_table(old, new):
    print('Old New Command')
    longest = max(len(command) for command in old.keys() | new.keys())
    print('-' * (15 + max(0, longest - 7)))
    for command in sorted(old.keys() | new.keys()):
        print_result(command, old.get(command), new.get(command))

print_table(old, new)

Run this with:

python3 compare.py results.tsv results2.tsv

This renders the following table with still failing in red, still passing in green, newly failing with a red background, and newly passing with a green background.

Old New Command

----------------------------------------------------

  0   0 ./bnorm-u8-via-binary-postops-cpp

  0   0 ./cnn-inference-int8-cpp

  0   0 ./cnn-training-bf16-cpp

  0   0 ./cnn-inference-f32-cpp

  0   0 ./cross-engine-reorder-cpp

  0   0 ./cnn-training-f32-cpp

  0   0 ./getting-started-cpp

  0   0 ./memory-format-propagation-cpp

  0   0 ./primitives-augru-cpp

  0   0 ./performance-profiling-cpp

  0   0 ./primitives-batch-normalization-cpp

  0   1 ./primitives-binary-cpp

  0   0 ./primitives-concat-cpp

  0   0 ./primitives-convolution-cpp

  0   0 ./primitives-eltwise-cpp

  0   0 ./primitives-inner-product-cpp

  0   0 ./primitives-layer-normalization-cpp

  0   0 ./primitives-lrn-cpp

  0   0 ./primitives-lstm-cpp

  0   1 ./primitives-matmul-cpp

  0   1 ./primitives-pooling-cpp

  0   0 ./primitives-prelu-cpp

  0   0 ./primitives-reduction-cpp

  0   0 ./matmul-perf-cpp

  0   0 ./primitives-reorder-cpp

  0   0 ./primitives-resampling-cpp

  0   0 ./primitives-shuffle-cpp

  0   0 ./primitives-softmax-cpp

  0   0 ./primitives-sum-cpp

  0   0 ./rnn-training-f32-cpp

  0   0 ./sycl-interop-buffer-cpp

  0   0 ./sycl-interop-usm-cpp

  0   0 ./tutorials-matmul-inference-int8-matmul-cpp

This achieves our original objective of performing a regression analysis on results from tests that were run in parallel.

Pair Programming with ChatGPT: Bash Retry with STDIN Replay

2023-04-07T01:00:00-04:00

ChatGPT can be a very useful programming assistant. I’ve used it for debugging, documentation reference, and more recently trying out random ideas I think I would find useful. Its conversational nature and correctibility make it conducive to problem solving.

Unreliable services outside of your control are disruptive to automation. Sometimes the best you can do is retry while praying that the problem goes away after a few attempts. It would be useful to be able to wrap routines that depend on such services in a retry function.

A basic formulation of the problem is relatively straightforward to implement, but a trickier aspect to handle is retrying commands that accept input on STDIN. Properly retrying such a command requires capturing and replaying what is received on STDIN. This added complication made it feel like a fun challenge to try using ChatGPT as a pair-programming assistant.

Before highlighting the progression of the effort, here is a direct link to the final implementation: retry.sh

Here’s the original problem statement I gave it:

Hello! I need a bash script that provides a function called retry. Its first argument is the number of times to retry a command, with 0 meaning to retry forever. The remaining arguments are a command and its arguments to retry. The tricky part about this retry function is that I need it to be able to have buffered what it receives on STDIN, such that it can replay it to the command on each retry. For example, if I write retry 4 bash -e < <(echo "date && false"), the inner command (bash -e) will receive the text date && false every time it’s retried, even though the retry function only received that text on its STDIN once.

It cheerfully produced the following output:

#!/bin/bash

retry() {
  local retries="$1"
  shift
  local cmd=("$@")

  local tmp_input_file
  tmp_input_file="$(mktemp)"
  cat >"${tmp_input_file}"

  local count=0
  local exit_code

  while [[ "${retries}" -eq 0 || "${count}" -lt "${retries}" ]]; do
    "${cmd[@]}" <"${tmp_input_file}"
    exit_code="$?"

    if [[ "${exit_code}" -eq 0 ]]; then
      break
    else
      count=$((count + 1))
    fi
  done

  rm -f "${tmp_input_file}"

  return "${exit_code}"
}

I was pleasantly surprised to find that it behaved correctly for the example I provided!

➜  ~ retry 4 bash -e < <(echo "date && false") 
Wed Apr  5 07:20:24 PM EDT 2023
Wed Apr  5 07:20:24 PM EDT 2023
Wed Apr  5 07:20:24 PM EDT 2023
Wed Apr  5 07:20:24 PM EDT 2023
➜  ~ echo $?
1

At this point I started wondering about a handful of corner cases, and how many of them it could help me address. The first one I pursued was not needing to buffer all of STDIN before starting the first attempt.

Is it possible to modify this to not need to wait for all of stdin to be received before running the command?

Its idea was to replace the temporary file with a named pipe fed by a background job, which implemented the behavior I specified. However, in doing so, it broke the ability for reattempts to receive the same STDIN data as the first attempt because data in named pipes is consumed when read.

@@ -5,25 +5,28 @@ retry() {
   shift
   local cmd=("$@")
 
-  local tmp_input_file
-  tmp_input_file="$(mktemp)"
-  cat >"${tmp_input_file}"
+  local fifo
+  fifo="$(mktemp -u)"
+  mkfifo "${fifo}"
+
+  cat >"${fifo}" &
 
   local count=0
   local exit_code
 
   while [[ "${retries}" -eq 0 || "${count}" -lt "${retries}" ]]; do
-    "${cmd[@]}" <"${tmp_input_file}"
+    "${cmd[@]}" <"${fifo}"
     exit_code="$?"
 
     if [[ "${exit_code}" -eq 0 ]]; then
       break
     else
       count=$((count + 1))
+      echo >&2 "Retrying... (${count}/${retries})"
     fi
   done
 
-  rm -f "${tmp_input_file}"
+  rm -f "${fifo}"
 
   return "${exit_code}"
 }

When I challenged it on this, it acknowledged the error and switched back from a named pipe to a temporary file (retaining the background job to feed it), and used tail to feed the contents of the file into the command.

Will this be able to replay everything that was received on stdin? I forgot if a named pipe can be read from multiple times without the data that has been read becoming consumed.

@@ -5,28 +5,30 @@ retry() {
   shift
   local cmd=("$@")
 
-  local fifo
-  fifo="$(mktemp -u)"
-  mkfifo "${fifo}"
-
-  cat >"${fifo}" &
+  local tmp_input_file
+  tmp_input_file="$(mktemp)"
+  
+  # Background process to write stdin to temporary file
+  cat > "${tmp_input_file}" &
 
   local count=0
   local exit_code
 
   while [[ "${retries}" -eq 0 || "${count}" -lt "${retries}" ]]; do
-    "${cmd[@]}" <"${fifo}"
+    # Read from the temporary file and feed it to the command
+    "${cmd[@]}" < <(tail -n +1 -f "${tmp_input_file}")
     exit_code="$?"
 
     if [[ "${exit_code}" -eq 0 ]]; then
       break
     else
       count=$((count + 1))
-      echo >&2 "Retrying... (${count}/${retries})"
     fi
   done
 
-  rm -f "${fifo}"
+  # Clean up background process and temporary file
+  wait
+  rm -f "${tmp_input_file}"
 
   return "${exit_code}"
 }

I also asked it about correctly handling interrupts such as Ctrl-C. It reasonably added a cleanup function to a trap.

Will this clean up the background cat process if interrupted with ctrl-c? Will the retry function itself exit on ctrl-c or a termination signal? It should.

@@ -10,6 +10,15 @@ retry() {
 
   # Background process to write stdin to temporary file
   cat > "${tmp_input_file}" &
+  bg_pid=$!
+
+  cleanup() {
+    kill "${bg_pid}" 2>/dev/null
+    rm -f "${tmp_input_file}"
+  }
+
+  # Trap to handle signals and clean up
+  trap cleanup EXIT INT TERM
 
   local count=0
   local exit_code
@@ -27,8 +36,7 @@ retry() {
   done
 
   # Clean up background process and temporary file
-  wait
-  rm -f "${tmp_input_file}"
+  cleanup
 
   return "${exit_code}"
 }

When I tested this version, I found that the script was not exiting when the command succeeds. I suspected the use of tail was responsible for this.

This works when the command to retry fails. When the command succeeds, the script hangs and never exits, which is undesirable. I think this is because the tail command is not terminating. I’m not sure why it terminates when the command fails, but not when the command succeeds.

Its next idea was to switch to using cat instead of tail, and to nest it in a loop to continuously read data from the temporary file.

@@ -25,14 +25,18 @@ retry() {
 
   while [[ "${retries}" -eq 0 || "${count}" -lt "${retries}" ]]; do
     # Read from the temporary file and feed it to the command
-    "${cmd[@]}" < <(tail -n +1 -f "${tmp_input_file}")
-    exit_code="$?"
-
-    if [[ "${exit_code}" -eq 0 ]]; then
-      break
-    else
-      count=$((count + 1))
-    fi
+    while ! "${cmd[@]}" < <(cat "${tmp_input_file}"); do
+      exit_code="$?"
+
+      if [[ "${exit_code}" -eq 0 ]]; then
+        break 2
+      else
+        count=$((count + 1))
+        if [[ "${retries}" -ne 0 && "${count}" -ge "${retries}" ]]; then
+          break 2
+        fi
+      fi
+    done
   done
 
   # Clean up background process and temporary file

This is a confusing change, as the value of having two nested loops is unclear. I observed that it never exits when the command succeeds, and reports success when a command fails.

This updated version results in strange behavior. echo "date && false" | retry 4 bash -e produces one line of output before exiting with status 0, when I expected four lines of output (for four attempts) and an exit status of 1. When I run echo "date && true" | retry 4 bash -e, it retries indefinitely and never returns.

The next attempt was to revert back to using tail, and depends on the cleanup function killing the background cat process, which is supposed to cause tail to exit.

@@ -25,18 +25,14 @@ retry() {
 
   while [[ "${retries}" -eq 0 || "${count}" -lt "${retries}" ]]; do
     # Read from the temporary file and feed it to the command
-    while ! "${cmd[@]}" < <(cat "${tmp_input_file}"); do
-      exit_code="$?"
-
-      if [[ "${exit_code}" -eq 0 ]]; then
-        break 2
-      else
-        count=$((count + 1))
-        if [[ "${retries}" -ne 0 && "${count}" -ge "${retries}" ]]; then
-          break 2
-        fi
-      fi
-    done
+    "${cmd[@]}" < <(tail -n +1 -f "${tmp_input_file}")
+    exit_code="$?"
+
+    if [[ "${exit_code}" -eq 0 ]]; then
+      break
+    else
+      count=$((count + 1))
+    fi
   done
 
   # Clean up background process and temporary file

This change also makes no sense, because the command blocking ensures that cleanup never runs, thus never killing the background cat process. I decided that I needed to help it out a bit, and proposed that the background process stream STDIN to the command line-by-line through a named pipe, exiting if it detects that the command running in the background has exited.

The false test works correctly, but the true test hangs after printing the expected one line of output. It seems like the tail command is still not terminating, preventing progression of the script from reaching the cleanup function that would kill the background cat command. Instead of using tail, another function may need to be introduced that reads from tmp_input_file, checking if the command receiving input has terminated or not in between each line read and sent. Maybe a named pipe could be used by the function that reads from tmp_input_file to stream stdin to the command? I’m not sure if this would work though, or if it’s the best approach.

The implementation it generated served as the starting point for my own manual iterations on the program.

@@ -1,5 +1,20 @@
 #!/bin/bash
 
+stream_input() {
+  local tmp_input_file="$1"
+  local cmd_pid="$2"
+  local fifo="$3"
+
+  while kill -0 "${cmd_pid}" 2>/dev/null; do
+    while read -r line; do
+      echo "${line}" > "${fifo}"
+      if ! kill -0 "${cmd_pid}" 2>/dev/null; then
+        break
+      fi
+    done < "${tmp_input_file}"
+  done
+}
+
 retry() {
   local retries="$1"
   shift
@@ -12,9 +27,14 @@ retry() {
   cat > "${tmp_input_file}" &
   bg_pid=$!
 
+  local fifo
+  fifo="$(mktemp -u)"
+  mkfifo "${fifo}"
+
   cleanup() {
     kill "${bg_pid}" 2>/dev/null
     rm -f "${tmp_input_file}"
+    rm -f "${fifo}"
   }
 
   # Trap to handle signals and clean up
@@ -24,8 +44,15 @@ retry() {
   local exit_code
 
   while [[ "${retries}" -eq 0 || "${count}" -lt "${retries}" ]]; do
-    # Read from the temporary file and feed it to the command
-    "${cmd[@]}" < <(tail -n +1 -f "${tmp_input_file}")
+    # Start the command
+    "${cmd[@]}" < "${fifo}" &
+    cmd_pid=$!
+
+    # Stream input to the command
+    stream_input "${tmp_input_file}" "${cmd_pid}" "${fifo}" &
+
+    # Wait for the command to finish
+    wait "${cmd_pid}"
     exit_code="$?"
 
     if [[ "${exit_code}" -eq 0 ]]; then

It was at this point that I decided I had reached the point of diminishing returns, wherein the program was becoming more complex than it was capable of usefully understanding. The fact that even after pointing it in a solid direction it generated a plainly broken implementation was good evidence of this (stream_input will repeatedly dump the contents of tmp_input_file into the named pipe).

The final implementation ended up being significantly more complicated than I was originally expecting this problem to require. It is included below with explanations.

retry.sh

#! /usr/bin/env bash

# usage: retry ATTEMPTS COMMAND [...]
#
# This script provides a retry function capable of capturing and replaying data
# from stdin to the command being retried, so that each attempt receives the
# input.  The data from stdin is streamed to the command immediately, it is not
# pre-buffered, and thus will not lock up if the input from stdin never ends.
#
# If ATTEMPTS is 0, COMMAND is retried forever until it succeeds.

# Suppress some of the job control messages.  Unfortunately it is impossible to
# suppress all of them.
set +m

retry() {
    local retries="$1"
    shift
    local cmd=("$@")

    # Background process which reads stdin data from stdin_reader through
    # in_fifo, buffers it in replay_file, and writes it to the stdin of each
    # command attempt through out_fifo.
    stream_input() {
        local cmd_pid="$1"
        local stdin_reader_pid="$2"
        local in_fifo="$3"
        local out_fifo="$4"
        local replay_file="$5"

        # Hold out_fifo open for writing.  The command might otherwise exit
        # prematurely if the pipe is closed by a bash I/O redirect operator in
        # between individual line writes.
        exec 6>"${out_fifo}"

        while read -r line; do
            echo "${line}" >&6
            if ! kill -0 "${cmd_pid}" 2>/dev/null; then
                # Stop replaying from replay_file to out_fifo if the command
                # has exited.
                break
            fi
        done < "${replay_file}"

        # This is used to signal if data is done being sent over in_fifo by
        # stdin_reader.
        local final_read=false

        exec 5<"${in_fifo}"

        while true; do
            if read -t 0 -r -u 5; then
                read -r -u 5 line
                echo "${line}" >> "${replay_file}"
                echo "${line}" >&6
            elif ${final_read}; then
                break
            fi

            if ! kill -0 "${cmd_pid}" 2>/dev/null; then
                # Stop passing lines from in_fifo to out_fifo if the command
                # has exited.
                break
            elif ! kill -0 "${stdin_reader_pid}" 2>/dev/null; then
                # Since stdin_reader has exited, nothing new will come over
                # in_fifo.  Signal the next iteration of the loop to finish
                # reading whatever remains in the pipe and exit.
                final_read=true
            fi
        done

        # Close out_fifo and in_fifo.
        exec 5<&-
        exec 6>&-
    }

    # Create a named pipe to receive data from stdin asynchronously. Bind the
    # pipe to &3 to keep it open on both ends.  It needs to be held open for
    # writing because subsequent instances of stream_input will block on
    # opening it for reading if stdin_reader has exited, which would otherwise
    # have been the only thing holding it open.  It needs to be held open for
    # reading to prevent holding it open it for writing from blocking.
    local in_fifo
    in_fifo="$(mktemp -u)"
    mkfifo "${in_fifo}"
    exec 3<>"${in_fifo}"
    cat > "${in_fifo}" &
    stdin_reader_pid=$!

    # The replay file buffers data received on stdin so that it can be passed
    # to reattempts of the command.
    local replay_file
    replay_file="$(mktemp)"

    cleanup() {
        # Kill all background jobs
        # https://unix.stackexchange.com/a/609300
        for j in $(jobs | awk '{gsub("[^0-9]","",$1);printf "%%%s\n", $1}'); do
            kill $j
        done
        [[ -e "${in_fifo}" ]] && rm -f "${in_fifo}"
        [[ -e "${replay_file}" ]] && rm -f "${replay_file}"
        [[ -e "${out_fifo}" ]] && rm -f "${out_fifo}"
        wait
    }

    trap cleanup EXIT INT TERM

    local count=0
    local exit_code

    while ((retries == 0 || count < retries)); do
        # Create a named pipe to receive buffered and replayed data from stdin
        # from stream_input.
        local out_fifo
        out_fifo="$(mktemp -u)"
        mkfifo "${out_fifo}"

        "${cmd[@]}" < "${out_fifo}" &
        cmd_pid=$!

        stream_input \
          "${cmd_pid}" \
          "${stdin_reader_pid}" \
          "${in_fifo}" \
          "${out_fifo}" \
          "${replay_file}" &
        stream_input_pid=$!

        wait "${cmd_pid}"
        exit_code="$?"

        rm -f "${out_fifo}"

        if ((exit_code == 0)); then
            break
        else
            count=$((count + 1))
        fi
    done

    cleanup

    return "${exit_code}"
}

Now that I have a functional implementation, it is easy to see in hindsight that bash was likely not the best choice for this. It’s if nothing else an interesting example of the types of limits you can run into when using ChatGPT for programming problem-solving.

You're Not Crazy: Dockerfile Build Caching for ADD/COPY is Broken

2023-03-20T01:00:00-04:00

Consider this workflow. You have created a git repository on your local machine containing a Dockerfile and a file it copies into the resulting image.

Dockerfile

# syntax=docker/dockerfile:1.5
FROM debian:bullseye
ADD cat-me cat-me
RUN cat cat-me

cat-me

mow!

You also add a simple GitHub Action that builds and pushes a cache of this image to a registry, so that developers checking out the repository and running the build do not need to waste time rebuilding parts that have already been built.

.github/workflows/build.yml

on:
  push:
    branches: [master]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: docker/login-action@v2
      with:
        registry: ghcr.io
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    - uses: docker/setup-buildx-action@v2
    - run: |
        docker buildx build \
          --pull \
          --cache-to type=inline \
          --cache-from type=registry,ref=ghcr.io/lynxqtel/cat:latest \
          --push \
          -t ghcr.io/lynxqtel/cat:latest \
          .

This uses buildkit to build and push an image with its cache inlined, which allows incremental step results to be reused in subsequent builds with --cache-from, both in the Actions workflow and on developers’ machines. Developers who have cloned the repository can leverage this automation to produce builds that can be downloaded as cache without needing to run the builds on their own machine. This can be extremely useful if there are multiple different variants of the image that could be built, the automation machines are significantly more powerful or parallelizable than the developers’ machine, or if the builds just take a very long time.

The above is a fanciful idea that is not true. The setup as described will not always produce cache hits when a developer runs a comparable build command.

User Groups and UMASK

A process’ umask controls the permissions for new files. On Ubuntu systems, /etc/login.defs provides its default definition and attributes as UMASK 022 and USERGROUPS_ENAB yes. The value of the umask masks (NANDs) which bits of the user, group, and other permission fields can be set. The file provides the following commentary on these settings:

# UMASK is the default umask value for pam_umask and is used by
# useradd and newusers to set the mode of the new home directories.
# 022 is the "historical" value in Debian for UMASK
# 027, or even 077, could be considered better for privacy
# There is no One True Answer here : each sysadmin must make up his/her
# mind.
#
# If USERGROUPS_ENAB is set to "yes", that will modify this UMASK default value
# for private user groups, i. e. the uid is the same as gid, and username is
# the same as the primary group name: for these, the user permissions will be
# used as group permissions, e. g. 022 will become 002.

What this means is for a developer cloning a repository under an account with a GID matching their UID (private user group), the permissions of all the files in the checkout will be -rw-rw--r-- due to the USERGROUPS_ENAB setting altering the umask from 022 to 002. If the developer’s account GID does not match their UID, the umask is left unmodified at 022, resulting in permissions of the checked-out files to be -rw-r---r--. This means that the group of an account being a private user group or not affects permissions for new files.

ADD/COPY Caching

Both the Docker daemon and the docker-container buildx driver builders compute a checksum of the file(s) in the build context specified in an ADD or COPY instruction. The files’ permissions are included in this checksum, and it is currently not possible to prevent different permissions from producing different checksums. Even when using the --chmod option, this only applies to the file after the checksum is computed, rather than before.

It is in this way that having a different effective umask between two checkouts will cause ADD/COPY instructions to produce different checksums, which can cause cache misses depending on its value on the machine that built and pushed the cache.

Solutions

There are no elegant ways to deal with this. It can either be addressed on the user side or the producer side. A user could use a script to manually set permissions of files in a Docker build context before performing a build, such as with a git checkout hook. Alternately, the producer can publish multiple cache builds of the same image for each variant of umask expected to be relevant across all the developers. In this case, additional --cache-from flags must be provided on the developer’s build command for each variant. While the latter approach avoids placing a burden on other developers to only run build command-lines as part of or adjacent to permission-fixing shell scripts, it will add additional complexity to what may already be a complciated tagging scheme. I slightly prefer the former method as straightforward setup instructions for a git checkout hook that unifies the permissions of the entire Docker build context will effectively solve the problem without any additional action or accommodation needed.

Until an official solution to this problem is developed and merged, developers performing Docker image builds should take care to check that the permissions on the file they wish to include in the build match the permissions used when creating caches. Git does not track file permissions, so even with a completely clean repository different developers can get different cache hit success rates simply due to whether or not their GID matches their UID, and in the process of trying to discover this come dangerously close to descending into madness.

Beyond the Sidewalk's End: The Internet without Section 230

2023-02-25T00:00:00-05:00

(1) No provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider.

(2) No provider or user of an interactive computer service shall be held liable on account of any action voluntarily taken in good faith to restrict access to or availability of material that the provider or user considers to be obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise objectionable, whether or not such material is constitutionally protected; or any action taken to enable or make available to information content providers or others the technical means to restrict access to material described in paragraph (1). (47 U.S.C. § 230(c))

With the Supreme Court set to rule on Google’s liability for algorithmically-promoted terrorism content under Section 230 (archive) of the Communications Decency Act, I began imagining what may become of the Internet without its protections. Occasionally during moments of exasperation and frustration when I see the horrors wrought when commoners (archive) are allowed access to computers with an Internet connection, I fantasize about an alternate reality. In this admittedly elistist fantasy, computers remained primarily tools of education, science and engineering, and the Internet looks more like it did before a small handful of websites dominated a majority of the traffic and content generation: a diverse collection of smaller, individual websites and semi-isolated networks with the creative control solely in the hands of the operators and direct members of the surrounding communities. The technical burden of publishing content on the Internet fell mostly to the individual creating it, rather than publishing platforms like YouTube and Facebook that do most of the technical work for you, at the cost of needing to abide by their rules and to be subject to their moderating discretion.

In short: you had to know what you were doing (archive).

With the survival of Section 230’s protections uncertain, a return to these norms may no longer be out of the question.

Without Recommendation Engines
------------------------------

If providers can be held liable for consequences of user exposure to content promoted by recommendation engines (“the algorithm,”) the sophistication of these mechanisms may become significantly hobbled, with them perhaps even being done away with altogether. Content on social media platforms may be presented only in chronological order. Questions would arise about what constitutes a recommendation engine; are search engines just a type of recommendation engine? Certain types of platforms like YouTube could become extremely difficult to use without users possessing foreknowledge of what specific content they want to access.

Section 230 has always protected the ability of online services and their users to highlight relevant, useful content from others, connecting people to the material they enjoy. Legal risk for recommending or organizing content would reduce useful services like showing the best job listings, listing the most relevant products, or displaying the most helpful videos of recipes, songs, or sources of news, entertainment and information.

——Google, Gonzalez v Google and the future of an open, free and safe internet (archive)

I think Google’s assessment of the possible outcomes of a ruling against them is reasonable. Even moreso, such a ruling could precede a wave of new lawsuits that may seek to further chip away at Section 230’s afforded protections. The following is speculation on how the topology of what we currently know as the Internet may change in response.

Everything or Nothing
---------------------

If platforms can no longer afford the risk of permitting their users from publishing their own content without either the strictest of moderation or a complete lack of moderation, they will split into several distinct types: unmoderated badlands where only illegal content is removed, private websites and networks, and Club Penguin (archive).

As documented in The Twenty-Six Words that Created the Internet (archive), a 1995 ruling by a New York state court judge against the ISP Prodigy established a precedent that a computer service “provider” can be considered liable as a “publisher” if it exercises any editorial control of the content it hosts, even when submitted by its users.

By actively utilizing technology and manpower to delete notes from its computer bulletin boards on the basis of offensiveness and “bad taste”, for example, PRODIGY is clearly making decisions as to content (see, Miami Herald Publishing Co. v. Tornillo, supra ), and such decisions constitute editorial control….PRODIGY’s conscious choice, to gain the benefits of editorial control, has opened it up to a greater liability than CompuServe and other computer networks that make no such choice.

——Stratton Oakmont, Inc. v. Prodigy Services Co., 23 Media L. Rep. 1794 (N.Y. Sup. Ct. 1995) (archive)

If Section 230’s protections become undone, it is reasonable to predict that this precendent will again become relevant and applicable.

Section 230 supports free expression online, enabling websites to offer a wide-range of information and allow a diversity of voices to be discovered and heard. Without Section 230, some websites would be forced to overblock, filtering content that could create any potential legal risk, and might shut down some services altogether. That would leave consumers with less choice to engage on the internet and less opportunity to work, play, learn, shop, create, and participate in the exchange of ideas online.

——Google, Gonzalez v Google and the future of an open, free and safe internet (archive)

Survival
--------

Without the assistance of major platforms to publish and promote your content, each website and Internet service effectively becomes its own publisher, of its own operator’s content, unless you opt to publish on a platform alongside offensive content. Websites will diversify, and so will the infrastructure hosting and connecting them, becoming tightly coupled with each other. Special interest communities will form around constructing and sharing common infrastructure, and only members of those communities will be able to publish on it.

Even worse, connectivity itself may become balkanized. The potential liability of serving problematic content will see not just the hosting infrastructure becoming highly localized but the connectivity infrastructure as well, requiring you to be both a member of the relevant communities to even access their content and sufficiently geographically adjacent to an access point, as peering between these communities will be limited.

An extremely prescient example of this has recently taken place, with the attempted (and ultimately failed) removal of the Kiwi Farms (archive) from the Internet. A long-running DDoS attack has been targeting the site for months. It has suffered frequent outages as a result, and its owner Joshua Moon has struggled (but succeeded) in keeping it online due to pressure on infrastructure providers to discontinue not just DDoS protection but also hosting and even layer 3 connectivity. Tier 1 providers such as CentryLink and Arelion blackholed the site’s IP address range; access to the site was occasionally dependent on who your ISP was. None of this was done at the request or order of any legal proceeding or judgement, it was exclusively private companies providing important infrastructure (archive) exercising their right to moderate the content they serve. Section 230 also protects this, as does the United States’ lack of net neutrality rules.

I consider Joshua’s perspective on this valuable because of the unique challenges he has faced with keeping his site reachable from the clearnet. Irrespective of any deserved condemnation of the legally-hosted content of this particular site and its operator, the normalization of lower-level infrastructure providers engaging in content moderation will inevitably affect the Internet’s topology.

The Internet is also very fragile. It has many moving parts and drives the politics of our world today. For this reason, it will die, and it will die very soon. Our big-I Internet is being broken apart and will soon become many little-i internets. Every large country or trade union will have its own local and strictly regulated internet. Connections between internets will occur on the vestigial remains of the big-I Internet, requiring a special business permits to access.

——Joshua Moon, Where the Sidewalk Ends: The Death of the Internet (archive)

I believe the topology fragmentation can become even more extreme than this. It’s not so unreasonable to speculate that fear of legal liability could produce similar results as does fear of negative PR and business loss (archive).

I may yet see my strange regressive fantasy come to pass.

Avoid CMD and PowerShell with a Visual Studio Dockerfile with Bash support

2023-02-03T20:50:00-05:00

Automation for projects targeting different operating systems is sometimes written twice: once in a Unix shell scripting language (normally bash) for Linux, BSD, and macOS, and once in either cmd or PowerShell for Windows. Most of the logic in these scripts will be equivalent, and therefore duplicated. Duplicated code is more difficult and error-prone to maintain when making updates.

However, there are a handful of distributions of Bash for Windows, which let you write (nearly) identical scripts across all targeted operating systems. GitHub Actions makes this easy by making these distributions of Bash available on the Windows runners.

I’ve created a Dockerfile that can serve as a base image for building software on Windows using Bash, both within a derivative image and executing as a development container. It is derived from an example from Microsoft. It installs the specified version of the Visual Studio Build Tools, the Chocolatey package manager, Ccache, 7zip, and Git (which includes a distribution of Bash when installed with the /GitAndUnixToolsOnPath parameter). It also sets the entrypoint to ...\vcvars64.bat && which ensures any command executed as part of a container run will do so with the 64-bit build tools available, including cmake.exe and cl.exe.

windows-toolchain.Dockerfile

# escape=`

FROM mcr.microsoft.com/windows/servercore:ltsc2019

SHELL ["powershell"]
RUN New-ItemProperty `
  -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" `
  -Name "LongPathsEnabled" `
  -Value 1 `
  -PropertyType DWORD `
  -Force
ARG VS_VERSION=2019
ENV VS_VERSION=${VS_VERSION}
RUN `
[System.Net.ServicePointManager]::SecurityProtocol = `
  [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; `
Switch ($env:VS_VERSION) { `
    "2019" {$url_version = "16"} `
    "2022" {$url_version = "17"} `
    default {echo "Unsupported VS version $env:VS_VERSION"; EXIT 1} `
}; `
wget `
  -Uri https://aka.ms/vs/${url_version}/release/vs_buildtools.exe `
  -OutFile vs_buildtools.exe

SHELL ["cmd", "/S", "/C"]
ENV VS="C:\Program Files (x86)\Microsoft Visual Studio\${VS_VERSION}"
RUN `
(start /w vs_buildtools.exe --quiet --wait --norestart --nocache `
  --installPath "%VS%\%VS_VERSION%\BuildTools" `
  --add Microsoft.VisualStudio.Workload.VCTools `
  --includeRecommended `
  || IF "%ERRORLEVEL%"=="3010" EXIT 0) && `
del /q vs_buildtools.exe && `
mklink /d "%VS%\current" "%VS%\%VS_VERSION%" && `
if not exist "%VS%\%VS_VERSION%\BuildTools\Common7\Tools\VsDevCmd.bat" exit 1

ENTRYPOINT ["C:\\Program Files (x86)\\Microsoft Visual Studio\\current\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat", "&&"]

SHELL ["powershell"]
ENV chocolateyUseWindowsCompression false
RUN `
Set-ExecutionPolicy Bypass -Scope Process -Force; `
[System.Net.ServicePointManager]::SecurityProtocol = `
  [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; `
iex ((New-Object System.Net.WebClient).DownloadString(`
  'https://community.chocolatey.org/install.ps1'))
SHELL ["cmd", "/S", "/C"]
RUN `
call "%VS%\current\BuildTools\VC\Auxiliary\Build\vcvars64.bat" && `
powershell -command `
  "[Environment]::SetEnvironmentVariable('Path', $env:Path, [System.EnvironmentVariableTarget]::Machine)"
RUN `
choco feature disable --name showDownloadProgress && `
choco install -y --fail-on-error-output git -params '"/GitAndUnixToolsOnPath"' && `
choco install -y --fail-on-error-output 7zip && `
choco install -y --fail-on-error-output ccache

Derivative image

When used as a base image in a derivative Dockerfile, bash.exe is an available shell. Because Windows Dockerfiles do not support extended Buildkit syntax features like heredocs, instructions must be strung together with &&.

# escape=`

FROM windows-toolchain:latest
SHELL ["bash.exe", "-x", "-e", "-c"]
RUN "`
curl -OSL --ssl-no-revoke https://zlib.net/zlib1213.zip && `
7z.exe x zlib1213.zip && `
rm zlib1213.zip && `
cd zlib-1.2.13 && `
mkdir build && `
cd build && `
cmake -G Ninja -D BUILD_SHARED_LIBS=YES .. && `
cmake --build . --target INSTALL && `
cd ../.. && `
rm -rf zlib-1.2.13"

Build environment

When used as a build environment, you can pass Bash code to be executed in with a Bash heredoc (<<EOF).

wget https://zlib.net/zlib1213.zip
unzip zlib1213.zip
rm zlib1213.zip
cd zlib-1.2.13
mkdir build
docker run \
  --rm \
  --interactive \
  -v $(cygpath -ad .):'C:\work' \
  -w 'C:\work\build' \
  windows-toolchain:latest \
  bash -e -x <<EOF
cmake -G Ninja -D BUILD_SHARED_LIBS=YES ..
cmake --build . --target INSTALL
EOF
cd ..
rm -rf zlib-1.2.13

Aside from the setup code in the windows-toolchain Dockerfile, no cmd or powershell code is necessary to compile code using Visual Studio Build Tools. This allows automation to be written once in Bash, with minimal conditions for handling operating system differences.

Eliminating Code Duplication in Dockerfiles with Stage Name Aliasing

2023-02-03T10:54:00-05:00

The limited availability of traditional control flow constructs when writing Dockerfiles encourages code duplication. This increases the odds of errors made in subsequent updates wherein a developer may update one of the copies but accidentally forget to update the other(s). One alternative to these is build arguments, which can be leveraged to make decisions about which stages are the sources of other stages in multi-stage builds. This allows you to design Dockerfiles with built-in conditionals, with each branch of the conditional being defined as its own stage, allowing shared code to be branch points instead of needing multiple copies of them in completely independent stages.

One application of this is choosing between compiling a dependency versus downloading a pre-built binary. Downloading a pre-built binary could be useful for testing that binary prior to release, and compiling it could be useful for debugging it on a local development machine. Including a stage capable of compiling it as the dependency for its runtime stage allows developers to share a standardized declarative build environment and toolchain, making the results reproducible between developers.

However, lack of support for build argument evaluation in COPY --from directives discourages consolidating equivalent logic across these branched stages into a single downstream stage. Here’s an example with a Dockerfile for jq which is alternately downloaded or built from source depending on a build argument, and then copies the binary into a runtime layer with only the dependencies required for execution.

# syntax=docker/dockerfile:1.5

ARG JQ_SOURCE=download

FROM debian:bullseye AS jq-build
RUN apt-get update && apt-get install -y build-essential libonig-dev
ADD https://github.com/stedolan/jq/releases/download/jq-1.6/jq-1.6.tar.gz \
  jq-1.6.tar.gz
RUN <<EOF
tar xf jq-1.6.tar.gz
cd jq-1.6
./configure
make
make install
EOF

FROM debian:bullseye-slim AS jq-build-run
RUN apt-get update && apt-get install -y libonig5
COPY --from=jq-build /usr/local/bin/jq /usr/local/bin/jq

FROM debian:bullseye AS jq-download
ADD https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
  /usr/local/bin/jq
RUN chmod +x /usr/local/bin/jq

FROM debian:bullseye-slim AS jq-download-run
RUN apt-get update && apt-get install -y libonig5
COPY --from=jq-download /usr/local/bin/jq /usr/local/bin/jq

FROM jq-${JQ_SOURCE} AS jq-run
ENTRYPOINT [ "/usr/local/bin/jq" ]

Both the jq-build and jq-download stages produce an executable at /usr/local/bin/jq that can be copied into dependent stages, with the jq-*-run dependent stages being duplicates of each other, with the only difference being the stage from which the binary is copied.

graph LR; jq-build -. COPY --from .-> jq-build-run; jq-download -. COPY --from .-> jq-download-run; jq-build-run -- FROM --> jq-run; jq-download-run -- FROM --> jq-run;

As documented in moby/moby#34482, variables are not evaluated when they appear in a COPY --from= value. An attempt to consolidate the jq-build-run, jq-download-run, and jq-run stages using a build argument directly would not work:

FROM debian:bullseye-slim AS jq-run
RUN apt-get update && apt-get install -y libonig5
COPY --from=${JQ_SOURCE} /usr/local/bin/jq /usr/local/bin/jq
ENTRYPOINT [ "/usr/local/bin/jq" ]

However, because build arguments are evaluated in FROM statements, a stage can be conditionally aliased to a common name, which can be used as a dependency in both subsequent FROM and COPY --from= statements. jq-source is used as an alias stage in this modified version.

# syntax=docker/dockerfile:1.5

ARG JQ_SOURCE=download

FROM debian:bullseye AS jq-build
RUN apt-get update && apt-get install -y build-essential libonig-dev
ADD https://github.com/stedolan/jq/releases/download/jq-1.6/jq-1.6.tar.gz \
  jq-1.6.tar.gz
RUN <<EOF
tar xf jq-1.6.tar.gz
cd jq-1.6
./configure
make
make install
EOF

FROM debian:bullseye AS jq-download
ADD https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
  /usr/local/bin/jq
RUN chmod +x /usr/local/bin/jq

FROM jq-${JQ_SOURCE} AS jq-source

FROM debian:bullseye AS final
RUN apt-get update && apt-get install -y libonig5
COPY --from=jq-source /usr/local/bin/jq /usr/local/bin/jq
ENTRYPOINT [ "/usr/local/bin/jq" ]

This produces a dependency graph with fewer stages and no code duplication.

graph LR; jq-build -- FROM --> jq-source; jq-download -- FROM --> jq-source; jq-source -. COPY --from .-> jq-run

Using the JQ_SOURCE argument as a conditional switch, this image can be built in two different ways:

docker buildx build --build-arg JQ_SOURCE=download - < Dockerfile
docker buildx build --build-arg JQ_SOURCE=build - < Dockerfile

In the first docker buildx build command, jq-source evaluates to jq-download, causing the COPY statement in the final stage to evaluate to:

COPY --from=jq-download /usr/local/bin/jq /usr/local/bin/jq

And likewise for the second build command:

COPY --from=jq-build /usr/local/bin/jq /usr/local/bin/jq

This demonstrates how aliasing stage names computed from build arguments can be used to implement sophisticated conditional logic about how different stages can depend on each other, preventing the need to write multiple Dockerfiles or to duplicate code within a Dockerfile.