pema.dev

Mipmap selection in too much detail

2025-05-09T00:00:00+00:00

In this post, I want to shed some light on something I’ve been wondering about for a while: How exactly are mipmap levels selected when sampling textures on the GPU? If you already know what mipmapping is, why we use it, and what pixel derivatives (ddx() / ddy()) are, you can skip to the section Derivatives to mipmap levels. The post does, however, assume some knowledge of graphics programming.

Mipmapping primer

A very common operation in shaders is texture sampling. In HLSL, we typically use the Texture2D.Sample() function for this. For a regular 4-channel floating-point texture, the full function signature looks like this:

float4 Texture2D.Sample(SamplerState sampler, float2 location);

It takes as input a location to sample the texture at, and a SamplerState, which is an object containing metadata used to influence the sampling, like which texture filtering mode should be used.

Here is an example of a simple fragment shader which shades each pixel by sampling a texture _Texture using the texture coordinates in the 0th UV channel:

float4 frag (float2 uv : TEXCOORD0) : SV_Target  
{  
    return _Texture.Sample(sampler_Texture, uv);  
}

When applied to a quadrilateral and fed a brick texture as input, we get this:

Notice that the surface looks rather ‘pixelated’, especially near the far edge. This is called texture aliasing. The root cause is that the size of the texture space region covered by a screen pixel varies with distance and viewing angle.

Consider, for illustration, the tilted quadrilateral. Observe how the same area in screen space can correspond to very different areas in texture space. The blue square covers roughly 4 pixels, while the green one covers roughly 12, even though they have the same area in screen space.

The more shallow the viewing angle, the more texels are covered by each pixel on the screen. Since we only sample 1 texel for every screen pixel, most of the texels covered by the screen pixel are aliased into a single sample - the contribution from the covered texels that were not sampled is lost!

The typical solution to this is mipmapping - we first generate a bunch of smaller versions of our texture, called mipmaps, by averaging texels from the higher resolution texture. As the mipmaps get smaller, their contents get more blurry. We are effectively applying a series of low-pass filters to the texture. We typically name the mipmaps using numbers, where 0 is the full resolution texture, and each subsequent number corresponds to a lower resolution texture. I’ll call these numbers ‘mipmap levels’. In cases where texture sampling would result in aliasing, because too many texels are covered by the screen pixel, we instead sample a lower resolution mipmap to mitigate it. That way, we get an approximate average of the texels in the covered area. Here’s the same setup as before, but now with mipmapping enabled: I didn’t have to change the shader at all to achieve this - when Texture2D.Sample() is used, and the input texture contains mipmaps, the GPU will automatically select an appropriate mipmap level for each sample!

If you are like me, you won’t find this explanation very satisfying. The GPU does some magic to select a mipmap level? Ok - so how does it work? That’s what the rest of this post is about.

Pixel derivatives

A common mental model for a fragment shader is “the piece of code inside of a big parallel for-loop iterating over every pixel on the screen”. The fragment shader is invoked for every pixel in parallel; each invocation gets a set of per-pixel inputs, and uses them to calculate the final color at that pixel. This is a fine mental model, but it doesn’t tell the full story.

In reality, fragment shaders aren’t invoked for every pixel in isolation - rather, they are invoked for 2x2 blocks of pixels, typically called “pixel quads”, or just “quads”. This setup allows us to approximate partial derivatives in screen space for any variable using the finite difference method! In HLSL, the functions to do this are called ddx() and ddy(), and return partial derivatives along the X and Y axes in screen space, respectively. You couldn’t implement these functions yourself - they are magic intrinsics which are implemented in hardware by subtracting the values associated with different pixels in the quad.

Now, observe what happens if we apply a small fragment shader that visualizes the partial derivatives along the Y-axis of the UV coordinates we were using to sample the texture from earlier:

float4 frag (float2 uv : TEXCOORD0) : SV_Target  
{  
    // Multiply by 15 so we can actually see the derivatives,  
    // they are typically very small.
    return float4(ddy(uv), 0, 1) * 15;
}

The more shallow the viewing angle, the larger the partial derivatives get. Additionally, the partial derivatives are larger further away from the camera.

These conditions line up well with the cases where texture sampling would result in ugly aliasing, so it shouldn’t come as a big surprise that the selection of mipmap levels has something to do with the partial derivatives of the input coordinates used to sample the texture. In fact, Texture2D.Sample() can be seen as syntax sugar for the more general Texture2D.SampleGrad() function:

// These 2 lines of code are equivalent:
Texture2D.Sample(sampler, location);
Texture2D.SampleGrad(sampler, location, ddx(location), ddy(location));

Texture2D.SampleGrad() takes the partial derivatives of the sampling location explicitly, whereas Texture2D.Sample() implicitly calculates them using ddx() and ddy(). Passing these explicitly can be useful in cases where the sampling locations are warped in a way that prevents ddx() and ddy() from returning useful partial derivatives (for example, during textured raymarching or parallax mapping). It also lets us use analytical partial derivatives in cases where we have a way to compute them.

This description still leaves an open question: How exactly does Texture2D.SampleGrad() use the input partial derivatives to determine which mipmap level to sample?

Derivatives to mipmap levels

Texture2D.SampleGrad() has some internal logic that maps from the partial derivatives of the sampling location to a specific mipmap level. This function directly corresponds to a hardware instruction on the GPU, though, so we unfortunately can’t just ‘look at the code’. To get a better idea of how it works, we can ask 2 questions: What does the mapping look like conceptually, and what does the hardware do in practice? Both questions are surprisingly difficult to find good answers to online, but I’ll tackle them both one by one.

What does the mapping look like conceptually?

One place you might think to look for information on how the mapping should work, is in the spec for your graphics library of choice. I find the GLES3.0 spec particularly readable. According to section 3.8.10.1 “Scale Factor and Level of Detail”, the mipmap level is calculated like so:

\[MipLevel(x, y) = log_2(\rho(x, y))\]

Where $x, y$ are the coordinates of the screen pixel, and $\rho$ is a “scale factor” defined by:

\[\rho = max\Bigg\{\sqrt{\bigg(\frac{\partial u}{\partial x}\bigg)^2 + \bigg(\frac{\partial v}{\partial x}\bigg)^2},\sqrt{\bigg(\frac{\partial u}{\partial y}\bigg)^2 + \bigg(\frac{\partial v}{\partial y}\bigg)^2}\Bigg\}\]

Here, $u$ and $v$ denote the coordinates of the sampling location in texture space, scaled to the range $[0; TextureSize]$, so $\frac{\partial u}{\partial x}$ for example is the partial derivative of the horizontal coordinate of the sampling location with respect to the $x$, i.e. ddx(u * textureSize).

That’s a bit of a mouthful, so let’s translate it to code:

float MipLevel(float u, float v, float textureWidth, float textureHeight)
{
    float du_dx = ddx(u * textureWidth);
    float dv_dx = ddx(v * textureHeight);
    float du_dy = ddy(u * textureWidth);
    float dv_dy = ddy(v * textureHeight);

    float lengthX = sqrt(du_dx*du_dx + dv_dx*dv_dx);
    float lengthY = sqrt(du_dy*du_dy + dv_dy*dv_dy);
    float rho = max(lengthX, lengthY);

    return log2(rho);
}

This should be somewhat intuitive - as the magnitude of the partial derivatives increase, so should the mipmap level. We only really care about the axis where the aliasing will be worst (the axis with the largest magnitude), hence the max(). The log2() accounts for the fact that each increment in mipmap level corresponds to sampling a texture that is half the size.

If you search around the web for how to calculate mipmap level manually, you will probably find a solution like this. However, as we will soon see, this is not the full story.

What does the hardware actually do?

Since Texture2D.SampleGrad() is implemented in hardware, we can’t look at the implementation directly. However, we can use observations of its behavior to deduce what it is doing. To facilitate this, I will first need a texture with mipmaps that are easily distinguishable from each other. Luckily, most graphics frameworks let you manually fill in the texture data for each mipmap - they don’t have to be blurry versions of the full-resolution texture. Since I’m using Unity for visualization, I wrote a little script that produces a texture where each mipmap is a single, bright, clearly distinguishable color. The resulting texture looks like this:

Next, I wrote a shader that samples the texture using Texture2D.SampleGrad(), but where the sampling location is a constant, and the partial derivatives are based on UV coordinates. Texture2D.SampleGrad() takes a partial derivative of the sampling location for both the X and Y axes, each of which are 2-dimensional vectors, so this is technically a 6-dimensional function. Since we are keeping the sampling location fixed, it is a 4-dimensional function in practice. Functions of over 3 dimensions are a bit tricky to visualize, so I’ll focus on just the partial derivatives for the X axis for now, and leave the Y axis partial derivatives as 0:

float4 frag (float2 uv : TEXCOORD0) : SV_Target  
{  
    // Scale UVs from [0; 1] to [-1; 1]
    float2 scaledUV = (uv - 0.5) * 2.0;

    // Sample texture at location (0.5, 0.5),
    // using X derivative = UV coords,
    // and Y derivative = 0.
    float2 samplingLocation = 0.5;
    float2 yDerivative = 0;
    float4 hardwareSample = _MainTex.SampleGrad(
        sampler_MainTex,
        samplingLocation,
        scaledUV.xy,
        yDerivative);  
  
    return hardwareSample;
}

Applying this shader to a quadrilateral gives this result on my machine (GPU is an RTX 4070 super): The image is scaled such that the bottom left corner corresponds to X-axis partial derivatives of (-1, -1), and the top right corner corresponds to (1, 1). The color at each pixel indicates which mipmap level is being sampled, when Texture2D.SampleGrad() is passed partial derivatives equal to the pixel coordinates.

When I first saw this result, I was a bit surprised. Why are there jagged edges!? The formulas described in the previous section should result in a bunch of perfect concentric circles! As it turns out, current graphics libraries leave the specifics of the implementation up to the GPU vendor, and only prescribe some loose criteria on the implementation. Nvidia cards use a particularly crude approximation.

Side note: This is just one of many pieces of functionality that differ in implementation across different vendors. It’s a pet peeve of mine when people assume that there is a “one true correct” implementation of pretty much anything in GPU-land. To name a few other areas where vendors differ: The precision of transcendentals like cos(x) and sin(x), the implementation of AlphaToCoverage (AMD uses dithering), subtle differences in rasterizer output, especially with conservative rasterization, etc. This is unfortunately a big source of pain when doing image-based/golden-image testing for anything involving the graphics pipeline, and is one of the big reasons such test setups often have per-platform reference images, and use fuzzy comparisons.

My next instinct was to determine which vendors actually differ in their implementation, and to what extent. I asked a bunch of friends to try it on their hardware and collected the results. I’m only rendering the quadrant with positive partial derivatives here, since the shape is symmetric anyway:

A few notes on my findings:

All tested vendors seem to roughly agree on the mapping function. Every image resembles a set of concentric circles with the same radius. The extent to which the circle is approximated differs quite a bit across every vendor, though.
No two vendors agree entirely on the implementation.
All tested vendors seem to be consistent across their hardware lineup. Interestingly, AMD’s APUs are consistent with their dedicated GPUs.
The result is the same across all the graphics libraries Unity supports (I checked).

And with this, I present perhaps the first-ever GPU tier list ranked in order of fidelity of partial-derivative-to-mipmap-level-mapping-function (the only important metric, clearly): Adreno/Qualcomm > AMD > Intel > MacBook/Apple > Nvidia. You heard it here first, Nvidia hardware bad! (/s)

Exploring the full 4D mapping function

In the previous visualizations of the partial-derivative-to-mipmap-level mapping function, I’ve only visualized the influence of X-axis partial derivatives and have left the Y-axis partial derivatives at (0, 0). Next, I’d like to visualize the influence of both axes simultaneously. The easiest way I found to visualize this 4-dimensional function is using a grid. In each grid cell, we will have an image where the X-axis partial derivatives vary. Each grid cell will use fixed values for the Y-axis partial derivative, the values depending on the location of the cell. This way, we can see 2-dimensional “slices” of the full function. I whipped up a Unity script to render that out using the shader from earlier, which produced this image:

Note: This kind of visualization will pop up several times throughout the post, so it’s worth hammering in what we are looking at: The bottom left grid cell has constant Y-axis partial derivatives (0, 0), while the top right grid cell has constant Y-axis partial derivatives (1,1), all the cells between have constant Y-axis partial derivatives somewhere between 0 and 1. Within each grid cell, the local X and Y coordinates determine the X-axis partial derivatives. I’m only visualizing positive Y-axis partial derivatives, because the images for negative partial derivatives are just the same, but mirrored. The color once again indicates the selected mipmap level. Of course, this image is very vendor-specific - these images were all generated on an RTX 4070 Super.

With this setup, we can easily compare what the hardware does to the proposed software implementation from the What does the mapping look like conceptually? section, using Texture2D.SampleLevel() to explicitly sample the mipmap level we have selected. Here’s the same image, but using the software implementation: That looks… quite different from the hardware implementation. As expected, the software implementation produces perfect circle patterns, while the hardware implementation approximates them. But additionally, the hardware implementation produces some kind of ellipse shape whenever both the X- and Y-axis partial derivatives have nonzero components, while the software implementation only ever produces perfect circles. This surprised me a bit when I first saw it, since I found no mention of this behavior during my initial research. This piqued my interest and drove me to attempt to reverse engineer the behavior. The answers were out there, I just had to dig a little deeper…

Reverse engineering the hardware implementation

The first resource I managed to find, which made mention of an ellipse in the context of mipmap level selection, was the DirectX 11.3 Functional Spec (praise be), in Section 7.18.11 “LOD Calculations”. I’ll briefly restate the relevant parts of the spec, then dissect them:

Given a pair of partial derivative vectors representing an elliptical transform, it is important to calculate LOD using a proper orthogonal Jacobian matrix, as described by Heckbert 89. When performing anisotropic filtering, it is also important to use these modified vectors to calculate the proper filtering footprint. D3D11.3 will allow approximations to this effect. The following describes the ideal transformation, given 2-dimensional vectors:

Implicit ellipse coefficients:

A = dX.v ^ 2 + dY.v ^ 2
B = -2 * (dX.u * dX.v + dY.u * dY.v)
C = dX.u ^ 2 + dY.u ^ 2
F = (dX.u * dY.v - dY.u * dX.v) ^ 2

Defining the following variables:

p = A - C
q = A + C
t = sqrt(p ^ 2 + B ^ 2)

The new vectors may be then calculated as:

new_dX.u = sqrt(F * (t+p) / ( t * (q+t)))
new_dX.v = sqrt(F * (t-p) / ( t * (q+t)))*sgn(B)
new_dY.u = sqrt(F * (t-p) / ( t * (q-t)))*-sgn(B)
new_dY.v = sqrt(F * (t+p) / ( t * (q-t)))

The following caveats also apply:
- if either of dX or dY are of zero length, an implementation should skip these transformations.
- if dX and dY are parallel, an implementation should skip these transformations.
- if dX and dY are perpendicular, an implementation should skip these transformations.
- if any component of dX or dY is inf or NaN, an implementation should skip these transformations.
- if components of dX and dY are large or small enough to cause NaNs in these calculations, an implementation should skip these transformations.
if(ComputeIsotropicLOD), the LOD calculation is:

float lengthX = sqrt(dX.u*dX.u + dX.v*dX.v + dX.w*dX.w)
float lengthY = sqrt(dY.u*dY.u + dY.v*dY.v + dY.w*dY.w)
output.LOD = log2(max(lengthX,lengthY))

Phew, that’s a bit of a mouthful… In the snippets from the spec, dX and dY represent the X- and Y-axis partial derivatives we’ve discussed previously. The w component of these vectors is only used for 3D- and cubemap textures, so it isn’t relevant here. Notice that the code in the very last snippet looks almost exactly like the software implementation we tested earlier. However, most of the text before that snippet describes some kind of ‘elliptical’ transformation that should be applied to the partial derivatives before we calculate their lengths. This sounds quite promising, so let’s try to grok what the spec is prescribing.

Missing elliptical transformation

Side quest: Texture filtering theory and vector calculus

Before I can describe the elliptical transformation, we need to visit a few ideas from the theory of texture filtering and from vector calculus. If you are already familiar with jacobians and the concept of pixel footprint, you can probably skip to the next section Understanding the elliptical transformation.

A naïve approach to rendering a textured surface is this: For each screen pixel, determine the corresponding point on the surface, find its corresponding point in texture space, then read the value in the texture nearest to this point. As we’ve established earlier, this approach causes aliasing; when a screen pixel covers too many texels, and only one texel is sampled, the contribution from most of the covered texels is lost. A better approach would be to integrate over the area covered by each screen pixel, gathering contribution from all the covered texels. Mipmapping is a cheap way to approximate this integral using precomputed tables.

Ideally, we’d like to integrate over the projection of the screen pixel onto the texture. This projection is called the “footprint” of the pixel. To do this, we want to know how a unit area in screen space (i.e., a pixel) is transformed when projecting into texture space. The typical mathematical tool for describing changes in area due to mappings between spaces, such as projection, is called “the jacobian matrix” or just “the jacobian”. When used to transform a point, the jacobian of the mapping provides the best linear approximation of the mapping at that point. When the jacobian is a square matrix, its determinant describes how much space is ‘stretched’ at that point. The jacobian is constructed from first-order partial derivatives of coordinates in the input space with respect to coordinates in the output space. In our case, the input space is screen space, and the output space is texture space. The jacobian looks like this:

\[\Large \begin{bmatrix} \frac{\partial u}{\partial x} & \frac{\partial u}{\partial y} \\ \frac{\partial v}{\partial x} & \frac{\partial v}{\partial y} \end{bmatrix}\]

Where $(u, v)$ are coordinates in texture space, and $(x, y)$ are coordinates in screen space. If you’ve paid attention so far, this might ring a few bells - these partial derivatives can be calculated in HLSL using ddx() and ddy(). In other words, we can view the partial derivatives passed to Texture2D.SampleGrad() as forming a jacobian which describes the projective mapping involved in rendering the textured surface.

In reality, it is impractical to integrate over the actual footprint of a screen pixel, as it will in general be a curvilinear quadrilateral (a quadrilateral with curved edges) - quite an unpleasant shape. Therefore, most approaches approximate it as either a regular quadrilateral or an ellipse, as illustrated in the figure below (Image source). When using mipmapping, we are mostly interested in the size of the footprint. The larger the area, the larger mipmap level we need to use, since higher mipmap levels correspond to approximations of integrals over larger regions of the texture.

Understanding the elliptical transformation

With these concepts in mind, let us dissect the elliptical transformation described in the DirectX 11 spec. The first paragraph reads:

“Given a pair of partial derivative vectors representing an elliptical transform, it is important to calculate LOD using a proper orthogonal Jacobian matrix, as described by Heckbert 89.”

I’ve highlighted the 2 important concepts with bold text. We have the partial derivatives of our texture sampling coordinates, but in what sense do these represent an elliptical transform? Imagine representing each screen pixel with a unit circle described by an angle $\theta$, such that $s(\theta)$ evaluates to all the points on the perimeter of the unit circle:

\[\Large s(\theta)= \begin{bmatrix} cos(\theta) \\ sin(\theta) \end{bmatrix}\]

If we additionally construct the jacobian described in the previous section:

\[\Large \begin{bmatrix} \frac{\partial u}{\partial x} & \frac{\partial u}{\partial y} \\ \frac{\partial v}{\partial x} & \frac{\partial v}{\partial y} \end{bmatrix}\]

… And transform the unit circle with the jacobian via matrix multiplication to get a new function:

\[\Large p(\theta) = \begin{bmatrix} \frac{\partial u}{\partial x} & \frac{\partial u}{\partial y} \\ \frac{\partial v}{\partial x} & \frac{\partial v}{\partial y} \end{bmatrix} \begin{bmatrix} cos(\theta) \\ sin(\theta) \end{bmatrix}\]

The resulting function $p(\theta)$ will describe an ellipse, where the 2 column vectors of the jacobian (i.e., the X- and Y-axis partial derivatives) are on the perimeter of the ellipse. In the special case where the column vectors are perpendicular and have the same length, the result is still just a (potentially scaled) circle. When the column vectors are perpendicular, but have different lengths, the result is an ellipse where the column vectors are the semi-major and semi-minor axes of the ellipse. If the vectors are not perpendicular, this doesn’t apply.

That explains the first part of the paragraph, so what is the “proper orthogonal Jacobian matrix” part about? When the column vectors of the jacobian are not perpendicular, the jacobian does not form an orthogonal basis - shapes transformed by the jacobian will be sheared. Recall that selecting a mipmap level involves calculating 2 lengths - in our software implementation from earlier, this was done like so:

float lengthX = sqrt(du_dx*du_dx + dv_dx*dv_dx);
float lengthY = sqrt(du_dy*du_dy + dv_dy*dv_dy);

When the column vectors of the jacobian are perpendicular, this code calculates the distance to the furthest point on the perimeter of an ellipse, whose semi-major and semi-minor axes are given by the column vectors. You can think of this as measuring how ‘stretched’ the ellipse is, along whichever axis is most stretched. That’s exactly what we want for mipmapping - the more stretched the elliptical footprint, the more texels it will cover, and the higher the mipmap level we need. However, when the column vectors are not perpendicular, the shearing breaks this stretch metric. We need to manually correct for this by calculating the metric in a different coordinate system. To do this, we need to diagonalize the ellipse.

Diagonalizing an ellipse means finding a coordinate system in which the ellipse is aligned with the axes. In particular, the major and minor axes should be aligned with the X and Y axis of the coordinate system. This eliminates the shearing, and allows us to calculate the stretch metric properly.

Note: For the curious reader, this ‘diagonalization’ is closely related to matrix diagonalization, which decomposes a matrix into its eigenvectors and eigenvalues. In fact, ellipses can be represented as a 2x2 matrix whose eigenvectors give the direction of the major and minor semi-axes, and whose eigenvalues are related to the length of the axes.

The next few paragraphs of the DirectX 11 spec describe an algorithm for performing this diagonalization, taken from Heckbert 89. The details of this algorithm are not so important, so I won’t describe them. Instead, to build intuition, I have implemented the algorithm in a graphing calculator, which lets us easily see what is going on:

Note: You can play with this interactive demo here.

The image shows an elliptical footprint corresponding to a case where the X- and Y-axis partial derivatives (labelled $d_x$, $d_y$) are not perpendicular. Notice how their length no longer describes the stretch of the ellipse. The outputs of the algorithm (labelled $n_x$, $n_y$) are a set of new basis vectors describing the transformation to a coordinate space in which the ellipse is axis-aligned. These new basis vectors are the column vectors of our “proper orthogonal Jacobian matrix”. This new jacobian describes almost the same mapping as the original one, and would produce the exact same ellipse as the original when used to transform the unit circle. The only difference is where the basis vectors are pointing.

This visualization also shows why the DirectX spec provides a bunch of corner cases where the diagonalization should not be applied:

The following caveats also apply:

if either of dX or dY are of zero length, an implementation should skip these transformations.

if dX and dY are parallel, an implementation should skip these transformations.

if dX and dY are perpendicular, an implementation should skip these transformations.

if any component of dX or dY is inf or NaN, an implementation should skip these transformations.

if components of dX and dY are large or small enough to cause NaNs in these calculations, an implementation should skip these transformations.

If either partial derivative is zero length, the coordinate would be flattened to an infinite line. The same happens if the partial derivatives are parallel: If the partial derivatives are perpendicular, there is no shearing, so performing the diagonalization is a waste of effort. Other than that, the final 2 points about inf and NaN are pretty self-explanatory.

With the elliptical transformation under our belt, let’s try adding it to our software implementation of mipmap level selection from earlier:

void EllipseTransformDerivatives(inout float2 dx, inout float2 dy)  
{  
    bool anyZero = length(dx) == 0 || length(dy) == 0;  
    bool parallel = (dx.x * dy.y - dx.y * dy.x) == 0;  
    bool perpendicular = dot(dx, dy) == 0;  
    bool nonFinite = isinf(dx) || isinf(dy) || isnan(dx) || isnan(dy);  
    if (!anyZero && !parallel && !perpendicular && !nonFinite)  
    {        
        float A = dx.y*dx.y + dy.y*dy.y;  
        float B = -2.0 * (dx.x * dx.y + dy.x * dy.y);  
        float C = dx.x*dx.x + dy.x*dy.x;  
        float F = (dx.x * dy.y - dy.x * dx.y) * (dx.x * dy.y - dy.x * dx.y);  
        float p = A - C;  
        float q = A + C;  
        float t = sqrt(p*p + B*B);  
        float2 newDx, newDy;  
        newDx.x = sqrt(F * (t+p) / ( t * (q+t)));  
        newDx.y = sqrt(F * (t-p) / ( t * (q+t)))*sign(B);  
        newDy.x = sqrt(F * (t-p) / ( t * (q-t)))*-sign(B);  
        newDy.y = sqrt(F * (t+p) / ( t * (q-t)));  
  
        bool failed = any(isnan(newDx) || isinf(newDx) || isnan(newDy) || isinf(newDy));  
        if (!failed)  
        {
            dx = newDx;
            dy = newDy;
        }
    }
}

Rendering out the same visualization we used before: This looks a lot more similar to what the hardware implementation was doing!

Trilinear filtering

All the visualizations I’ve made thus far have been without any kind of hardware texture filtering, and just use simple nearest-neighbor interpolation. Enabling bilinear filtering makes no difference, as it doesn’t affect how the mipmap level is selected. Enabling trilinear filtering, however, does make a difference. Rendered with the correct software implementation: And with hardware mipmap level selection: Trilinear filtering is just like bilinear filtering, but where we additionally interpolate between 2 mipmap levels. Without trilinear filtering, the selected mipmap level is just rounded to the nearest integer, and only one mipmap is sampled. With trilinear filtering, the floor and ceiling of the mipmap level are calculated, the 2 corresponding mipmaps are sampled, and the results are linearly interpolated using the fractional part of the mipmap level. This is why we’ve been calculating it as a float all along.

With that, harsh transitions between mipmap levels from the previous images are changed to smooth gradients. This is not very surprising, but I wanted to show that the software implementation continues to hold up when using trilinear filtering.

Anisotropic filtering

In addition to bilinear and trilinear filtering, most GPUs also support anisotropic filtering. Anisotropic filtering aims to alleviate one of the main issues with mipmapping. Mipmapping reduces aliasing from texture samples, but also introduces blurring, especially at shallow viewing angles. Anisotropic filtering effectively removes the blur by taking multiple texture samples from a higher resolution mipmap when the footprint of a pixel is unevenly stretched (Image source): The typical algorithm for anisotropic filtering works like this:

Determine the elliptical footprint of the screen pixel using the X- and Y-axis partial derivatives of the sampling location in texture space.
Determine the semi-major and semi-minor axes of the ellipse. The semi-major (longer) axis is called the “axis of anisotropy”.
Calculate the ratio of the lengths of the semi-major and semi-minor axes. This is called the “ratio of anisotropy”.
Clamp the ratio of anisotropy to the maximum degree of anisotropy - this is typically a setting on the texture, with possible values ranging from 0 to 16. The maximum degree is sometimes called the “aniso level”.
Calculate $log_2(semiMinorAxisLength)$. This is the mipmap level to sample.
Distribute sample points along the axis of anisotropy. The number of sample points depends on the clamped ratio of anisotropy.
Sample the texture at each of the sample points and average their values.

Notice that, unlike with regular mipmapping, where we use the length of the semi-major axis to select the mipmap level, we now use the length of the semi-minor axis. This is what mitigates the blurring caused by mipmapping - we sample a higher resolution texture when the pixel’s footprint is anisotropic (stretched differently on each axis). That alone would just reintroduce aliasing, so to mitigate this, we take more samples along the axis where they are needed most, effectively mitigating the added aliasing by supersampling, rather than blurring.

So, how does enabling anisotropic filtering change the visualization of mipmap level selection from before? To visualize this, I’ve upgraded our previous visualization to an animation. Since anisotropic filtering is parameterized by a maximum degree of anisotropy setting, the visualization is now an animation. Each maximum degree produces a different image: When the maximum degree of anisotropy exceeds 0, the image no longer matches what our software implementation produced. This makes sense - anisotropic filtering affects mipmap level selection, but our implementation isn’t accounting for it.

Returning to the DirectX 11 spec, recall that the paragraph about the elliptical transform read: “[…] it is important to calculate LOD using a proper orthogonal Jacobian matrix, as described by [Heckbert 89]. When performing anisotropic filtering, it is also important to use these modified vectors to calculate the proper filtering footprint.”. I skipped over this part earlier, but it is exactly what we are missing. The spec further elaborates: “if(ComputeAnisotropicLOD), the LOD calculation is:”

// Compute outputs:
// (1) float ratioOfAnisotropy
// (2) float anisoLineDirection
// (3) float LOD

float squaredLengthX = dX.u*dX.u + dX.v*dX.v
float squaredLengthY = dY.u*dY.u + dY.v*dY.v
float determinant = abs(dX.u*dY.v - dX.v*dY.u)
bool isMajorX = squaredLengthX > squaredLengthY
float squaredLengthMajor = isMajorX ? squaredLengthX : squaredLengthY
float lengthMajor = sqrt(squaredLengthMajor)
float normMajor = 1.f/lengthMajor

output.anisoLineDirection.u = (isMajorX ? dX.u : dY.u) * normMajor
output.anisoLineDirection.v = (isMajorX ? dX.v : dY.v) * normMajor

output.ratioOfAnisotropy = squaredLengthMajor/determinant

// clamp ratio and compute LOD
float lengthMinor
// maxAniso comes from a Sampler state.
if (output.ratioOfAnisotropy > input.maxAniso)
{
    // ratio is clamped - LOD is based on ratio (preserves area)
    output.ratioOfAnisotropy = input.maxAniso
    lengthMinor = lengthMajor/output.ratioOfAnisotropy
}
else
{
    // ratio not clamped - LOD is based on area
    lengthMinor = determinant/lengthMajor
}

// clamp to top LOD
if (lengthMinor < 1.0)
{
    output.ratioOfAnisotropy = MAX(1.0, output.ratioOfAnisotropy*lengthMinor)
}

output.LOD = log2(lengthMinor);

I’ve omitted some comments for brevity. The spec states that this code should run after applying the elliptical transformation from earlier. This is a pseudocode implementation of (part of) the algorithm for anisotropic filtering described earlier. It produces the ratio of anisotropy (ratioOfAnisotropy), the axis of anisotropy (anisoLineDirection), and the mipmap level (LOD).

To illustrate this, I’ve extended my interactive ellipse diagram from earlier with an implementation of the algorithm. As the partial derivative vectors move around, the axis of anisotropy remains the major axis of the ellipse, and the selected mipmap level depends on the size of the minor axis.

Note: You can play with the updated version of the diagram here.

For now, only the portions of the pseudocode from the DirectX spec relating to mipmap level calculation are relevant. This pseudocode is straightforward to translate into real shader code. Doing so yields the following: Nice - we seem to have a pretty decent match with the hardware implementation…

There is one noteworthy difference, however. On my GPU, every other value for the max degree of anisotropy barely looks different from the previous one. For example, Degree=4 and Degree=5 look very similar, though there is a slight difference. Then Degree=5 and Degree=6 look completely different. In the software implementation, we don’t have any such behavior. I’m not quite sure what is going on here, and it is probably extremely vendor-specific.

Bonus: More visualizations

To get a better feel for how the final software implementation of mipmap level selection compares to the hardware, I wrote a small shader that uses the software implementation on the left half of the screen, and the hardware implementation on the right. First, it can visualize which mipmap levels are chosen. Here’s with no filtering: And here is with 16x anisotropic filtering: Looks pretty good to me! Finally, we can visualize the actual effect of using these implementations on a typical textured surface: The halves are barely distinguishable to me! This is without anisotropic filtering, though. Unfortunately, we can’t visualize the difference with anisotropic filtering, as there is no way to manually specify the axis and ratio of anisotropy in shader code.

However, we can implement anisotropic filtering entirely in shadercode, by manually doing multiple samples. An implementation may look something like this:

// Get mip level, aniso line direction, and ratio of anisotropy
float ratioOfAnisotropy;
float2 anisoLineDirection;
float mipLevel = CalculateLodLevel(
    _MainTex_TexelSize.zw, ddx(i.uv), ddy(i.uv),
    _MaxAnisoLevel, ratioOfAnisotropy, anisoLineDirection);

// Take multiple samples and average them
if (ratioOfAnisotropy > 1.0)
{
    ratioOfAnisotropy = ceil(ratioOfAnisotropy);
    float4 sum = 0;
    for (float anisoIdx = 0; anisoIdx < ratioOfAnisotropy; anisoIdx++)
    {
        // Convert ratio to [0; 1] space
        float2 ratioInUVSpace = ratioOfAnisotropy * _MainTex_TexelSize.xy;

        // Get start and end positions of the anisotropic line
        float2 startPos = i.uv - anisoLineDirection * (ratioInUVSpace / 2.0);
        float2 endPos = startPos + anisoLineDirection * ratioInUVSpace;
        
        // Get the current sample position
        float2 samplePos = lerp(startPos, endPos, anisoIdx / (ratioOfAnisotropy-1));

        // Accumulate the sample
        sum += _MainTex.SampleLevel(sampler_MainTex, samplePos, mipLevel);
    }
    return sum / ratioOfAnisotropy;
}
else
{
    return _MainTex.SampleLevel(sampler_MainTex, i.uv, mipLevel);
}

Where CalculateLodLevel is an implementation of the pseudocode from the DirectX 11 spec. This implementation isn’t particularly good, and it doesn’t quite seem to match what the hardware does (the implementation details of anisotropic filtering are very much up to the vendors), but it does have the intended effect:

On this image, we see 3 setups side-by-side. On the left, we have no mipmapping. As expected, it looks very aliased. In the center we have software anisotropic filtering. It looks a lot less aliased, especially at shallow angles. Finally, on the right, we have regular mipmapping, with no anisotropic filtering. It has no aliasing, but is quite blurry.

Bonus: Nvidia’s approximation

I noted in the section What does the hardware actually do? that Nvidia uses a particularly crude approximation for their mipmap level selection. I can only assume they do this for performance reasons. I found their approximation quite intriguing, so I attempted to reverse engineer it. I believe they are doing something similar to this:

// Scale input derivatives to texture size  
dx *= float2(width0, height0); 
dy *= float2(width0, height0);

// Elliptical transform
EllipseTransformDerivatives(dx, dy);

// Absolute value to mirror around X and Y axis
float2 sx = abs(dx);
float2 sy = abs(dy);

// This is a cheap way to calculate the distance to a weird looking octagon
const float magic = 1.0/3.0;
float lengthX = lerp(sx.x+magic*sx.y, sx.y+magic*sx.x, step(sx.x, sx.y));
float lengthY = lerp(sy.x+magic*sy.y, sy.y+magic*sy.x, step(sy.x, sy.y));

// Regular logarithmic length -> mip mapping
float rho = max(lengthX,lengthY);  
float mipLevel = log2(rho);

That octagon distance check compiles down to just a few instructions:

ge r0.xy, v0.ywyy, v0.xzxx
and r0.xy, r0.xyxx, l(0x3f800000, 0x3f800000, 0, 0)
mad r1.xyzw, v0.yxwz, l(0.333333, 0.333333, 0.333333, 0.333333), v0.xyzw
add r0.zw, -r1.xxxz, r1.yyyw
mad o0.xy, r0.xyxx, r0.zwzz, r1.xzxx

And in particular, has no square roots or exponents. Rendered out: It doesn’t quite match the hardware implementation, but it’s pretty close. Close enough that I think I am on to something. If you think you have a better idea, let me know! For completeness, here is the diff between the hardware and my attempt rendered out:

Closing words

I hope that I’ve lived up to the title of this blog post, and that you learned a thing or two. I certainly did while researching this. Before we finish, I’d like to briefly touch on my motivation for writing this - it is twofold:

I initially became interested in how mipmap level selection works on GPU hardware when I was looking into writing a library for transpiling and running shaders on the CPU. There are many challenges involved in writing such a library, but one of the more annoying ones is deciphering how various hardware-implemented techniques can be done in software, based on often limited information. Mipmap level selection is one good example of this, but there are many others (think MSAA, rasterization rules, conservative rasterization, early Z, block texture compression, tessellation, etc.). This post explores just one of these rabbit holes.

What pushed me to dig deeper than the first software implementation I stumbled upon was this: I find myself frustrated with the relative lack of readily available information on GPU functionality specifics. So much of what I know comes from experimenting, exchanging ideas with fellow graphics programmers, and trying to decipher obtuse morsels of information drip-fed to me by the GPU vendors, through various documentation and pseudo-specs. I wish the details were easier to find, and this post is one small contribution towards fulfilling that. I hope you enjoyed following along.

Haskell is not category theory

2023-02-01T00:00:00+00:00

What, why, who?

If you have hung out in Haskell communities, you might have heard people mentioning concepts from the mathematical field of category theory, perhaps even claiming that Haskell is built upon category-theoretic foundations.

There is some truth to this, but I think it is more accurate to say that certain abstractions in Haskell are inspired by category theory. When learning Haskell, I took this connection too literally and ended up confusing the hell out of myself. This post is aimed at my former self and aims to clear up some confusions I came across while trying to make sense of it all.

Before we get started, I want to get a few important disclaimers out of the way:

This is not a Haskell tutorial and will assume some familiarity with the language. However, if you are proficient with any functional language, especially if it is a dialect of ML, you should be able to follow along.
You do not need to know any category theory to learn and become proficient with Haskell. If you are looking to learn the language, I recommend Learn You A Haskell.
I’ll try not to assume you know any category theory already.
I am not a category theorist. I consider myself a novice. This is just my attempt of making sense of the parts I am familiar with, so take it with a grain of salt.
As with several of my posts in the past, this is going to be structured more like a collection of notes, than as a coherent story.

What is a category?

Note: If you know any category theory, you can probably skip this section.

The primary object of study in category theory is the category. A category consists of:

A collection of objects.
A collection of morphisms, also known as arrows, which map from one object to another. I’ll write these as $f: a \rightarrow b$. Here, $f$ is an arrow from the object $a$ to the object $b$.
A binary operation called composition, which combines arrows together. I’ll write composition as $x \bullet y$ where $x$ and $y$ are arrows.

A few additional criteria must be satisfied:

Whenever we have an arrow from $a$ to $b$, and an arrow from $b$ to $c$, there is also an arrow from $a$ to $c$ which is the composition of the two. Thus, $(x: b \rightarrow c) \bullet (y: a \rightarrow b) = (z: a \rightarrow c)$.
Composition is associative, so $x \bullet (y \bullet z)$ = $(x \bullet y) \bullet z$.
For every object $a$, there is an identity arrow $I_a: a \rightarrow a$ which just maps from the object to itself.
Composition with an identity arrow has no effect, so $I \bullet f = f \bullet I = f$.

This is intentionally pretty abstract. In the next section, I’ll give a concrete example of a category.

$Hask$ is a category

A concrete example of a category is the category of Haskell types, typically called $Hask$. In this category:

Our objects are types, such as Boolean, String, [a], and so on.
Our arrows are just plain old functions. They take a value of one type and map it to a value of another.
Our composition operator is just function composition ‘.’.
The identity arrow for any type is the identity function, id.

All the criteria for a category are satisfied:

If we have a function x :: b -> c and a function y :: a -> b, then x . y will be a function of type a -> c.
Function composition is associative, (x . y) . z == x . (y . z).
Composing the identity function with any other function has no effect, id . f == f . id == f.

When we are talking about Haskell from a category-theoretic lens, we are almost always talking about $Hask$. Since types are more or less sets of possible values, this category is closely related to the category of sets, typically denoted $Set$.

$Hask$ is not a category

Note: If you are not interested in extreme pedantry, feel free to skip this section as well.

I lied. $Hask$ is technically not a category. There are several reasons for this, but I’ll focus on just one of them.

Haskell has the notion of a bottom value (uwu). Bottom values are a member of any type. A bottom value is trivial to define - we just recurse infinitely!

bottom :: a
bottom = bottom

We can use it in place of any type:

addNums :: Num a => a -> a -> a
addNums a b = a + b

-- no problems here!
addNums bottom 3

This value is allowed to be a member of any type because we can never fully evaluate it, so we will never actually violate any type constraint!

Haskell is a lazily evaluated language but provides a mechanism for strict evaluation through the seq combinator. The expression seq a b evaluates b, but will force a to be strictly evaluated.

With all of this in mind, let’s revisit our criteria for a category. It must hold that composing the identity function with any other function has no effect - id . a == a. Consider the following Haskell program:

foo :: a -> b
foo = foo

main :: IO ()
main = do
    print (seq (id . foo) 1337)
    print (seq foo 1337)

This program will output:

1337
<<loop>>

So we can tell the difference between foo and id . foo, breaking the criteria! Oh, and by the way <<loop>> is just Haskell’s way to tell us “this is definitely going to loop forever”.

`Functor` is not $Functor$

Another common object in category theory is a functor. A functor is a mapping (arrow) from a category to a category. I’ll write these as $F: A \rightarrow B$ where $F$ is a functor from category $A$ to category $B$. Such a functor will map:

Each object $a$ in category $A$ to an object $F(a)$ in category $B$
Each arrow $x: a \rightarrow b$ in category $A$ to an arrow $F(x) : F(a) \rightarrow F(b)$

In the above, $F(a)$ should be read as “the result of applying functor F to element a”, if $a$ is an object, we get an object back, and if $a$ is an arrow, we get an arrow back. You can view functors as a kind of polymorphic function on elements (be they objects or arrows) in a category.

A simple example of a functor is the identity functor, which does … wait for it … absolutely nothing! Every object and arrow in $A$ is mapped to the same object and arrow in $B$. Of course, an identity functor from $A$ to $B$ can only exist if $A$ and $B$ are the same since the mapping must be valid for all objects in $A$. We call functors from a category to that same category endofunctors.

Haskell’s standard library includes a typeclass (like a trait in Rust, or a C# interface on steroids) called Functor. It looks like this:

-- Functor typeclass declaration
class Functor f where
  fmap :: (a -> b) -> f a -> f b

Many built-in types implement this typeclass, such as lists and Maybe’s (optional values):

fmap (+2) [1, 2, 3] == [2, 3, 4]
fmap (*5) (Just 3) == Just 6
fmap (-10) Nothing == Nothing

If you are anything like me when I was learning Haskell, you might be tempted to assume that Haskell’s Functor is equivalent to a functor from category theory, but in fact, it is something much more restricted. Perhaps the most obvious difference is that every Haskell Functor is really an endofunctor on $Hask$, since $Hask$ is the only category we can talk about in Haskell. What other differences are there, though?

Recall that objects in $Hask$ are types, and arrows are functions. A proper functor $F$ would then have to consist of:

A mapping from a type a to a type F(a)
A mapping from functions of type a -> b to functions of type F(a) -> F(b)

Looking at the definition of fmap, this pretty clearly implements the second of the 2 mappings. But what about the first one? This would have to be a function taking a type as input, and returning a type as output - but Haskell cannot treat types as values directly, so we cannot write such a function!

Note: It is only partially true that Haskell has no type-level functions. With GHC extensions such as TypeFamilies, we can get something like this.

Instead, we’ll have to settle for the next best thing - type constructors. We can view a datatype like this:

data Foo a = Bar a

As a sort of type-level function from any type to a subset of all types. Specifically those with the shape forall a. Foo a. This is the missing mapping - the type constructors implementing the Functor typeclass.

To illustrate why this makes Haskell’s Functor a more limited abstraction than a full-blown category-theoretic endofunctor, let’s try to implement the dead simple identity functor mentioned earlier in this section. This is the best we can do:

data Identity a = Identity a

instance Functor Identity where
    fmap f (Identity a) = Identity (f a)

With a property identity functor $F$, we would have that that $F(a) = a$ for any object or arrow $a$. This clearly isn’t the case here; the values Identity "foo" and "foo" have two different types. Here we have an example of an endofunctor that cannot be made an instance of the Functor typeclass.

A note on kinds of functors

Another piece of information that is implicit in Haskell’s Functor typeclass, is that it is covariant. Covariance means that the direction of arrows in the category upon which the functor acts ($Hask$ in our case) is preserved. We can also define a contravariant functor, which flips the direction of arrows. In Haskell, the Contravariant typeclass is used to describe such functors:

class Contravariant f where
  contramap :: (a -> b) -> f b -> f a

Notice how contramap is almost equivalent to fmap, but that the direction of arrows (functions) is flipped. A function of type a -> b becomes a function of type F(b) -> F(a).

Haskell’s standard library contains several other kinds of functors lifted from category theory:

Bifunctor is a combination of 2 covariant functors, kind of like a pair Functor’s.
Profunctor is a combination of a covariant functor and a contravariant functor. Interesting for their use in lenses.
Applicative is essentially a monoidal functor. You may know that every Monad is also Applicative, and Applicative is a kind of weaker monad, which is what makes this abstraction interesting. Alas, these are beyond the scope of this post.
Probably even more that I don’t know of…

Natural transformations

I’ve described how functors are mappings between categories. Before moving on to monads, I should also mention mappings between functors. These are known as natural transformations. I’ll write these as $N_a : F(a) \rightarrow G(a)$ where $N$ is a natural transformation from functor $F$ to functor $G$. $N_a$ is called the component of $N$ at $a$. This is the concrete arrow of $N$ that applies to object $a$.

We can express natural transformations quite directly in Haskell - they are just polymorphic functions. Here’s an example using our Identity functor implementation from earlier:

identityToMaybe :: Identity a -> Maybe a
identityToMaybe (Identity a) = Just a

This particular function is a natural transformation from the Identity functor to the Maybe functor. The components of such a natural transformation are concrete instantiations of the polymorphic function, such as:

-- "component at Boolean"
identityToMaybe :: Identity Boolean -> Maybe Boolean

-- "component at [Integer]
identityToMaybe :: Identity [Integer] -> Maybe [Integer]

`Monad` is almost $Monad$

For the same reason that Haskell’s Functor is not quite a full-blown category-theoretic functor, Haskell’s Monad can never be full-blown. It comes a tad closer than in the case of functor, though.

In category theory, a monad is typically defined in terms of an endofunctor $F: C \rightarrow C$ together with 2 natural transformations $r: I_C \rightarrow F$ and $j: (F \times F) \rightarrow F$ where $I_C$ is the identity functor from $C$ to $C$, and $F \times F$ is the functor composition of $F$ with itself. Just as we could use $\bullet$ to compose 2 arrows into a new arrow, we can use $\times$ to compose 2 functors into a new functor. Additionally, a few monad laws must hold:

\[j \bullet Fj = j \bullet jF\]
\[j \bullet Fr = j \bullet rF = I_F\]

Where $I_F$ is the identity natural transformation from $F$ to $F$.

The notation $Fj$ means a natural transformation whose components $(Fj)_a$ are given by $F(j_a)$, ie. selecting the specific arrow from $j$ which applies to object $a$, and then applying functor $F$ to the result. Similarly, $jF$ is a natural transformation with components $(jF)_a = j_{F(a)}$, ie. applying the functor $F$ to object $a$ first, and then using the resulting object to select an arrow from $j$.

Let’s take a specific monad, Maybe, and see if we can write an implementation of $r$ and $j$ that seems to make sense. In the context of $Hask$, the natural transformation $r$ should be a function from the Identity functor to some other Functor. Since we are using Maybe as an example, this Functor is just Maybe. We get the following:

r :: Identity a -> Maybe a
r (Identity a) = Just a

Since Identity a carries no more information than a, let’s get rid of it:

r :: a -> Maybe a
r a = Just a

This function lets us lift a value of any type into our monad.

Next, recall that the natural transformation $j$ mapped from some composition of functors $F \times F$ to functor $F$. In Haskell, we would write $F \times F$ as Functor f => f (f a), or Maybe (Maybe a) for the concrete case of Maybe. We get the following:

j :: Maybe (Maybe a) -> Maybe a
j (Nothing) = Nothing
j (Just (Nothing)) = Nothing
j (Just (Just a)) = Just a

This function lets us remove one level of monadic structure, flattening a nested monad in a meaningful way. For the Maybe monad, it works sort of like a logical and.

With these implementations in mind, let’s take a look at how Haskell’s Monad typeclass looks (omitting a few irrelevancies):

class Applicative m => Monad m where
  (>>=) :: m a -> (a -> m b) -> m b
  return :: a -> m a

You can pretty easily convince yourself that the r function is precisely equivalent to the return function on this typeclass. But what about the j function? It is nowhere to be found. This confused me quite a bit when I was first learning about monads. It turns out that this missing function does exist in the standard library - just not on the Monad typeclass itself. This function is called join:

join :: Monad m => m (m a) -> m a

The missing piece of the puzzle is some nifty identities, which relate join and >>= (also known as the “bind” operator) together:

-- We can write >>= in terms of join and fmap:
(m >>= f) == (join (fmap f m))

-- Or join in terms of >>= and the identity function:
(join m) == (m >>= id)

Using this, we could write a valid instance of Monad for Maybe as so:

instance Monad Maybe where
    m >>= f = j (fmap f m)
    return = r

That’s all fine and good, but what about the 2 monad laws from earlier? Does our implementation satisfy these? Let’s take another look at the first law, $j \bullet Fj = j \bullet jF$, and translate each piece of the equation into Haskell:

The composition operator $\bullet$ is function composition ‘.’.
The natural transformation $j$ is the j function from earlier.
The $F$ construct in $Fj$ translates to fmap. We are applying our functor $F$ to the natural transformation $j$, which in Haskell was just a function. Remember, fmap is the portion of a functor that maps functions to functions.
The $F$ construct in $jF$ disappears. Here, the functor $F$ is applied to the objects (types) in $Hask$, turning any type a into the type Maybe a. We use this type to select the component of $j$ to apply. Concretely, we select the instantiation of j which applies to Maybe values - this is just j.

Putting all of this together, we can translate the law into Haskell like so:

j . fmap j == j . j

Cool. Does it hold? Well, let’s check every possibility:

(j . j) (Just (Just (Just 42))) == Just 42
(j . j) (Just (Just Nothing)) == Nothing
(j . j) (Just Nothing) == Nothing
(j . j) Nothing == Nothing

(j . fmap j) (Just (Just (Just 42))) == Just 42
(j . fmap j) (Just (Just Nothing)) == Nothing
(j . fmap j) (Just Nothing) == Nothing
(j . fmap j) Nothing == Nothing

Yep, it checks out! Following similar logic, we can translate the second of the 2 laws into Haskell as:

j . fmap r == j . r == id

Which also holds for all possibilities. Both (j . fmap r) and (j . r) will do absolutely nothing when applied to a Maybe value.

Monoid in the category of endofunctors

You might have heard the meme-worthy phrase “a monad is simply a monoid in the category of endofunctors” before. I don’t intend to give a full explanation of what this phrase means, but I’d like to build some intuition. Most of the words in this phrase should be familiar by now, but I haven’t yet defined what a monoid is. The classical definition of a monoid is:

A set S
… equipped with a binary operator $\otimes$
… which is associative, $(a \otimes b) \otimes c = a \otimes (b \otimes c)$
… and has an neutral element $ne$ such that for all $a \in S$, we have $a \otimes ne = ne \otimes a = a$

Typical examples include:

The set of natural numbers with $+$ as the operator, and $0$ as the neutral element.
The set of natural numbers with $*$ as the operator, and $1$ as the neutral element.
The set of strings with string concatenation as the operator, and “” as the neutral element.

In the last section, we derived a few laws for monads, written in Haskell:

j . fmap j == j . j
j . fmap r == j . r == id

Using the aforementioned standard library functions, we can generalize these to work for any monad:

join . fmap join == join . join
join . fmap return == join . return == id

If you stare at these laws for a moment, you might notice that they look sort of similar to the requirements for a monoid.

The first law is roughly saying “If we have a stack of 3 nested monadic values, it doesn’t matter if we first use join to flatten the innermost 2 values (the left-hand side), or the 2 outermost values (the right-hand side)”. In other words, it doesn’t matter how we group our join’s - this is the essence of associativity.

The second law is saying “If we inject an extra monadic layer into a monadic value using fmap return, and then flatten using join (left-hand side), or we wrap a monadic value in another monadic layer using return, and then flatten with join (middle part of the equation), we have achieved nothing. Since we achieved nothing, both of these operations are equivalent to the identity function id”. Here, return plays the role of a monoidal neutral element, and the law describes something similar to left and right identity.

This intuitive explanation should hopefully give a hint as to why we can consider monads to be a specific kind of monoid.

Don’t break the law

So far, I’ve shed some light on the differences between some common abstractions in Haskell, and their category-theory equivalents. I’ve been insinuating that the Haskell counterparts of these abstractions are proper subsets of category-theory concepts. However, for each of the abstractions I’ve described so far, this hinges on the adherence to some laws, such as the monad laws described in the previous section.

Haskell isn’t capable of enforcing these laws! For all of these abstractions, we can write implementations that type check but have absolutely nothing to do with category theory.

To illustrate this principle, let’s take a look at some properties of category-theoretic functors that I chose to omit earlier. For any functor $F: A \rightarrow B$, the following laws must hold:

$F(I_a) = I_{F(a)}$ where both $I$’s are identity arrows in their respective categories. Put simply, functors must preserve identity.
$F(x \bullet y) = F(x) \bullet F(y)$, for all arrows $x$ and $y$ in category $A$. Put simply, functors must preserve composition.

In Haskell, these laws are spelled:

fmap id == id
fmap f . fmap g == fmap (f . g)

Now consider the following botched implementation of the Functor typeclass for Maybe:

instance Functor Maybe where
    fmap a b = Nothing

This type checks just fine. Let’s check if the first law holds:

fmap id (Just a) == Nothing
id (Just a) == Just a

Oh no! It doesn’t. Interestingly, the second law does actually hold, because both sides of the equation just evaluate to a function that always returns Nothing.

Of course, these kinds of botched implementations are not common and are considered buggy code, but they are expressible, further distancing Haskell from a category-theoretic foundation.

Some links

6 interesting languages and their selling points

2022-07-29T00:00:00+00:00

Foreword

I’ve long been obsessed with everything relating to programming language design and implementation. I make an effort to seek out and try languages with novel features. In this post, I’d like to briefly present and attempt to sell you on some lesser-known, but very interesting languages I’ve come across. I won’t go very in-depth with any of them, but instead, try to demonstrate why you might want to use them. This is not only an excuse for you to try out some new tools but also an excuse for me to properly learn to use them. The languages aren’t in any particular order, so feel free to jump straight to whatever intrigues you using the links below.

Futhark - Effortless, functional GPGPU
ISPC - A programming model focused on vectorization
Koka - Algebraic effects
Unison - A language without source code
APL - Terseness taken to the extreme
Prolog - Logic programming

Futhark

Selling point: Effortless, functional GPGPU

Disclaimer: My opinion of Futhark is probably a bit biased since I recently had the pleasure of contributing to the compiler.

Programming for the GPU is a notoriously difficult and laborious discipline. Modern GPUs are strange beasts, very unlike CPUs. Writing efficient GPGPU (General Purpose GPU) code is a daunting task, often requiring a breadth of knowledge about the target architecture.

Futhark is a purely functional, data-parallel language that attempts to make programming for the GPU much more accessible and to unburden the programmer of much of the mental overhead usually implied by the paradigm.

In the case of Futhark, I think the best way to introduce the language is simply to show some code. Below is a Futhark program containing a single function, vector_length, which given an array of floats representing an n-dimensional vector, returns the length of that vector. It does this by first squaring each element, then summing all the elements up, and finally taking the square root of the sum.

def vector_length [n] (vec: [n]f32) : f32 =
  let squares = map (**2) vec
  let sum = reduce (+) 0.0 squares
  in f32.sqrt sum

Running this piece of code on the GPU, in all its parallel glory, is as simple as:

> futhark opencl test.fut
> echo [1, 2, 3] | ./test
3.741657f32

If you’ve ever had the displeasure of writing non-trivial CUDA or OpenCL code by hand, you’ll immediately notice that this code is basically the antithesis of how GPGPU usually looks. Futhark isn’t magic, though, and won’t automatically make your code go sanic fast. You have to play its game to see the gains.

Futhark conforms to the language paradigm typically known as array programming, roughly meaning that arrays are the primary object of focus, and the language is built to make operating on said arrays convenient and efficient. All parallelism in the language is expressed by a small handful of “array combinators”, which are higher-order functions, which take arrays and functions as input. If you’ve ever written in another functional language, or heck, even JavaScript, you may already be familiar with these:

-- Map takes as input a single-argument-function and an array, and
-- returns the result of applying that function to each element of
-- the array.
-- This example multiplies each element by 2.
map (*2) [1, 2, 3, 4] == [2, 4, 6, 8]

-- Reduce (aka. fold) takes as input a function implementing an
-- associative binary operation, a neutral element, and an array,
-- and returns the result of placing the binary operator between
-- each pair of elements in the array. This 'reduces' the array
-- to a single value.
-- This example thus calculates the sum of the passed array.
reduce (+) 0 [1, 2, 3, 4] == 10

-- Scan is exactly like reduce, except that it returns an array
-- of each of the intermediate values. This is sometimes known
-- as a prefix sum. 
-- This example returns an array with the values
-- [1, 1+2, 1+2+3, 1+2+3+4].
scan (+) 0 [1, 2, 3, 4] == [1, 3, 6, 10]

The Futhark compiler knows how to generate efficient, parallel GPU code implementing these combinators. Since all parallelism is derived from these, if your program does not use them, it will not be parallel, and probably quite slow - Futhark does not use a parallelizing compiler. This isn’t quite as limiting as it may sound, and the Futhark standard library contains a wide variety of functions that internally make use of these combinators. In fact, it is quite difficult to do anything useful in the language without making use of them.

Of course, just using combinators such as these isn’t enough to make more complex programs run fast. For this reason, the Futhark compiler is aggressively optimizing. In addition to the usual optimizations such as inlining and constant folding, the Futhark compiler implements some wacky domain-specific optimizations such as fusion and moderate flattening, which rearrange, transform, and sometimes eliminate array combinators so as to generate more efficient code. The result is that the code generated for any non-trivial Futhark program tends to look absolutely nothing like the code it was generated from.

-- The simplest form of "fusion" just turns 2 nested map's into a
-- single map with the operators passed to each composed:
map (+1) (map (*2) [1, 2, 3]) == map (\x -> (x * 2) + 1) [1, 2, 3]

Lastly, it is worth pointing out that Futhark is a truly purely functional language, unlike other so-called pure languages, such as Haskell, which claim to have no side effects, yet provide escape hatches out of the purity. After all, there is no such thing as IO in the absence of effects. Consequently, Futhark does not support IO. Instead, the language is built to be embedded into, and called from, a host language. The compiler can emit either Python or C code that uses CUDA or OpenCL internally. It is then your job to call into this generated code.

If you ever find yourself wanting to do some performant number-crunching, and your problem can be expressed elegantly in a functional language, definitely give Futhark a shot.

ISPC

Selling point: A programming model focused on vectorization

Again, I am quite biased on this language. I wrote my thesis about it, after all.

Moore’s Law is well known among programmers. Once upon a time, Moore postulated that the number of transistors in a CPU would double about every 2 years. For a long time, this conjecture seemed very reasonable, but as time passed, it veered further and further from reality. At some point, you simply can’t pack transistors any closer to each other without hitting a wall imposed by the laws of physics.

In our never-ending quest for better performance, multicore CPUs were invented somewhere along the way; if we can’t make a single processor any faster, why not just make more processors, and solve many problem instances at once? Programming languages were adapted to allow making use of this setup, usually via multithreading. Later still, vector instructions were introduced with instruction sets like SSE and AVX, which allowed processing multiple pieces of data once on a single core, in a paradigm dubbed SIMD (Single-Instruction-Multiple-Data).

Unfortunately, modern languages still make such vector instructions tedious and cumbersome to use, as most languages were never built around or fundamentally changed to afford the ergonomic use of SIMD. The options are essentially to invoke individual vector instructions manually via vector intrinsics, which are pretty painful to use and annoyingly tied to a single instruction set, or to rely on the autovectorizing capabilities of most modern compilers, which are brittle and difficult to reason with (Autovectorization is not a programming model). I’ve met a plethora of programmers who neglect SIMD entirely for these reasons, which is a shame since it is basically free performance.

Enough history. The ISPC language is an attempt to design a language specifically tailored for easy and ergonomic use of vector instructions with predictable performance. It does this by implementing a programming model dubbed SPMD (Single-Program-Multiple-Data). As the name implies, the programmer writes code that describes the actions to perform on a single piece of data, and the program is then implicitly run on multiple pieces of data. If you’ve ever written a shader in your life, this will probably feel familiar.

int sumArray(int arr[], int size) {
    int accum = 0;
    for (int i = 0; i < size; i++) {
        accum += arr[i];
    }
    return accum;
}

Consider the above C program, which sequentially sums the elements of an array. For the sake of explanation, let’s assume that our C compiler cannot automatically vectorize this code (although in reality, it probably can, since the code is so simple). In ISPC, an idiomatic analogous program may look like this:

uniform int sumArray(uniform int arr[], uniform int size) {
    varying int accum = 0;
    foreach (i = 0 ... size) {
        accum += arr[i];
    }
    return reduce_add(accum);
}

Although similar, the ISPC program will make use of vector instructions, and thus be several times faster. The first major differences are the uniform and varying qualifiers. ISPC supports 2 kinds of data, as annotated by these. Conceptually, ISPC code is executed by several “program instances” simultaneously, which are sort of analogous to threads. uniform indicates that a piece of data is shared between all program instances, while varying data may differ between each program instance.

Unlike threads, these conceptual program instances are always synchronized and correspond directly to scalars within a vector register. If we, for example, increment a varying variable using the += operator, this will automagically use a vectorized addition instruction, adding to each program instance’s variable simultaneously. As you might be able to guess by now, varying values are stored in vector registers.

The next difference, the foreach construct, is a special kind of loop that automatically distributes iteration across the various program instances, by splitting the iteration space into chunks. Thus, the loop index i is a varying value. In the first iteration of the loop, i will have a value of 0 for the first program instance, 1 for the second, 2 for the third, etc. If we, for example, assume 8 total program instances, in the next iteration i will be 8 for the first program instance, 9 for the second, 10 for the third, etc. If our total array size is 16, and the number of program instances is 8, this would mean the loop runs for 2 iterations, processing 8 elements simultaneously in each of the 2.

With all of that out of the way, let me give a brief explanation of what the program in the previous snippet does. First, we initialize a varying accumulator to all 0’s. Then, we loop over the array in a vectorized fashion. In each iteration, we load a vector of values from the array, and add it component-wise to the accumulator. After the loop, the accumulator will contain a sum per program instance. We then sum up each program instance’s contribution using the built-in function reduce_add, which takes as input a varying value, and returns a single uniform value. This is the final sum.

With relatively few changes to the original C program, we have produced a vectorized ISPC program, which is very readable - no intrinsics nonsense to be seen. This is a general theme for the language. Many C programs can be sped up immensely by lazily rewriting them in ISPC, replacing some for’s with foreach.

> ispc -h test.h -o test.o test.ispc
> gcc test.h test.c test.o -o exe
> ./exe

Like Futhark, ISPC isn’t intended for use as a standalone language. Instead, given an ISPC source file, the compiler emits a C/C++ header and an object file, which can then be linked into some existing C/C++ code, as shown with the commands above. This setup makes the language extremely easy to integrate into existing C/C++ codebases - in fact, it is used in Unreal Engine!

Koka

Selling point: Algebraic effects

In functional programming, we often talk about “(side) effects” and our desire to keep them under control. The definition of a side effect will vary a bit depending on who you ask, but I’ll give it a shot. A pure function is referentially transparent, which means that any call to the function can be replaced directly with its result, without changing the behavior of the program. If we call foo(5), and that evaluates to 10, then any and all occurrences of foo(5) can be replaced with 10. If the function does anything to break this property - by exhibiting side effects - it is no longer pure. Common examples of side effects are reading/writing to a filesystem or console, mutating global state, use of random number generation, etc.

In the absence of side effects, the behavior of code becomes simpler to reason with, and entire classes of bugs can be eliminated. As such, many functional languages have some kind of strategy for tracking and/or limiting side effects. A common approach is to use monads as seen in Haskell, PureScript, Scala, etc. Without going into much detail, they can be thought of as a design pattern for easily composing computations that exhibit side effects, using only pure functions. They are nice since they don’t necessarily have to be designed into the language, but can be implemented in any language with an expressive enough type system, such as F#, OCaml, and TypeScript, neither of which are purely functional.

Unfortunately, monads have some problems. They are notoriously intimidating to learn about for first-timers, and they can be tricky and tedious to combine. If you, for example, had a monad representing operations that perform IO, and a monad representing operations that may fail, and you wish to represent an operation that may fail and perform IO, you are often forced to either write tedious boilerplate code or use monad transformers to “stack the monads on top of each other”, which imo. can lead to some rather inelegant code.

Koka is a research language that takes a completely different approach to effect handling, embedding effects directly into the type system, and implementing a type of control flow typically known as algebraic effects.

I must interject here. Koka has a lot more cool features going for it than just algebraic effects. Stuff like Perceus refcounting, local mutability, and an insanely cool approach to syntax sugar. I’m focusing on algebraic effects since they are what I find most innovative about the language.

In Koka, each function type consists of 3 parts: The argument types, the result type, and the effect type. Below are a few examples from the documentation.

fun sqr    : (int) -> total int     // total: mathematically total
fun divide : (int,int) -> exn int   // exn: may raise an exception
fun turing : (tape) -> div int      // div: may not terminate
fun print  : (string) -> console () // console: may write to stdout
fun rand   : () -> ndet int         // ndet: non-deterministic

If we look at the divide function, we see that the 2 argument types are both integers, the result type is an integer, and the effect type is exn, denoting computations that may produce exceptions. Importantly, the effect type can be inferred, so you do not need to manually annotate it. The language can simply figure out for you which kind of side effects are being used in the function body.

Unlike monads, effect types can very easily be combined laterally. You can view the effect type of a function type as simply a list of all the effects used by the function. The syntax for combined effect types looks like this:

// This function may diverge, throw an error, or return an integer
fun foo() : <div, exn> int

In addition to the effect types built into the language, the programmer may define their own effect types. For example, let’s define an effect type that represents the ability to write messages to a log:

effect log
  fun write(msg: string) : bool

The name of this effect is log, and it enables exactly one operation - writing to a log via the write function. We make this function return a boolean to indicate success or failure.

Now, let’s write a simple program that does some integer arithmetic to demonstrate our new effect:

fun divide(a: int, b: int)
  a / b

fun add(a: int, b: int)
  a + b

fun main()
  val foo = add(3, 5)
  val bar = divide(foo, 2)
  println(bar)

This program will print the value 4. No effect shenanigans happening yet. Let’s say we want to add logging to our program, in order to keep track of which arithmetic operations we made during execution. We could use the write operation of our previously defined log effect, like so:

effect log
  fun write(msg: string) : bool

fun divide(a: int, b: int)
  write ("Dividing " ++ show(a) ++ " by " ++ show(b))
  a / b

fun add(a: int, b: int)
  write ("Adding " ++ show(a) ++ " by " ++ show(b))
  a + b

fun main()
  val foo = add(3, 5)
  val bar = divide(foo, 2)
  println(bar)

Notice how we didn’t have to change any function signatures. The compiler will automatically figure out that divide and add have the log effect type, although we could write it explicitly if we wanted. Compiling this code as is will result in an error:

error: there are unhandled effects for the main expression
  inferred effect : <console,test/log>
  unhandled effect: test/log
  hint            : wrap the main function in a handler

Koka is telling us that we are making use of an effect (log), but haven’t described how to handle this effect. In other words, we never gave an implementation for write. Helpfully, the compiler even tells us what to do. Enter effect handlers. Let’s change our main function up a bit:

fun main()
  with handler
    fun write(msg)
      println(msg)
      True
  val foo = add(3, 5)
  val bar = divide(foo, 2)
  println(bar)

This with handler construction lets us specify implementations of the effectful operations used in the rest of the function body. In this case, we just implement write as a simple print to stdout, and return True. Running the program now will produce the output:

Adding 3 by 5
Dividing 8 by 2
4

If the body of our function used more effects than just log, we could add their implementations under our implementation of write. What’s interesting about this code is that we can trivially swap the implementation of write used at the call site without having to change any other code. The code making use of the effect (add, divide) knows nothing about how that effect is implemented. For example, let’s swap our simple implementation with one that builds up a string instead of immediately printing, and which returns False on empty log entries:

fun main()
  var theLog := ""
  with handler
    fun write(msg)
      if msg == "" then
        False
      else
        theLog := theLog ++ "\n" ++ msg
        True
  val foo = add(3, 5)
  val bar = divide(foo, 2)
  println(bar)
  println("The log was: " ++ theLog)

Note the use of a mutable variable. Koka is arguably not purely functional, as it allows local mutable variables. They cannot escape the lexical scope, though. This code outputs:

4
The log was: 
Adding 3 by 5
Dividing 8 by 2

As we can see, the implementation of effects used by a piece of code is truly up to the caller of said code. The effectful code is generic, not bound to any specific underlying implementation. This is part of what makes algebraic effects so interesting and powerful. We can use them to establish a sort of 2-way “conversation” between caller and callee. Algebraic effects are in my opinion much simpler to understand and use than monads, and can be used to solve many of the same problems. They are also very flexible and can be used to implement basically any kind of side-effect or custom control flow, including but not limited to async/await, exceptions, global mutable state, IO, coroutines, etc.

As a final example, let’s return to our program above and implement our own error handling system - currently, we can divide by 0! First, we define a new effect type:

effect error<a>
  ctl error(msg: string) : a

Here, we use this more general ctl keyword for defining our effect’s single operation. Unlike fun, which acts exactly like a regular function, ctl lets us control when and whether the execution flow is passed between caller and callee. Since we don’t intend to return execution flow to the callee in the case of an error, the error operation can return an arbitrary, generic type a. Next, we change up our implementation of divide:

fun divide(a: int, b: int)
  if b == 0 then
    error("Don't divide by 0, dummy")
  else
    write ("Dividing " ++ show(a) ++ " by " ++ show(b))
    a / b

divide will now have the type (int, int) -> <error,log> int we can even use the Koka REPL to verify this:

> :t divide
check  : interactive
(a : int, b : int) -> <error,log> int

Finally, we simplify our main function a bit for testing:

fun main()
  with handler
    fun write(msg)
      println(msg)
      True
  println(divide(5, 0))

As before, if we compile the program as-is, we get an error about not handling the error effect. So, we add another with handler construction:

fun main()
  with handler
    fun write(msg)
      println(msg)
      True
  with handler
    ctl error(msg)
      println("Error: " ++ msg)
  println(divide(5, 0))

Which finally produces the output:

Error: Don't divide by 0, dummy

The use of ctl means that, by default, control flow never resumes at the callee. We can change that by calling the built-in resume operation.

fun main()
  with handler
    fun write(msg)
      println(msg)
      True
  with handler
    ctl error(msg)
      println("Error: " ++ msg)
      resume(42)
  println(divide(5, 0))

Which in this case lets us, at will, resume execution at the callee, passing a default value for the error call to evaluate to. This program produces:

Error: Don't divide by 0, dummy
42

I hope by now I’ve managed to convince you that algebraic effects are cool. I hope we see the feature in more languages in the future. They are currently a fairly common topic in programming language research but haven’t quite yet made it to the mainstream. Functional programming is, imo, not about eliminating side effects, but rather about structuring them in a reasonable manner, which lets us easily reason with the behavior of a program, avoiding nasty surprises from unexpected side effects. Algebraic effects are one cool way to achieve this.

As I alluded to in the intro of this section, Koka has a lot more going for it than just algebraic effects. I highly recommend you check out the language if you are interested in trying out cutting-edge features that push the current boundaries of functional languages.

Unison

Selling point: A language without source code

I must admit, the sentence I’ve written as the selling point of this language isn’t quite accurate. The Unison language does, in fact, have source code - it just isn’t what you store in your codebase. At first glance, Unison is a purely functional language in the ML family, with features that are reminiscent of many other, more well-known languages. What Unison does differently than any other language I’ve encountered is to “content-address” code.

To quote the Unison devs, “Each Unison definition is identified by a hash of its syntax tree”. A Unison codebase is not stored on disk as mutable text files containing source code but as a dense database of definitions addressable by these hashes. The idea seems a bit wacky at first, but yields several nice benefits over other languages, some of which I’ll describe shortly.

The language is a bit tricky to demonstrate, as the programmer’s workflow relies quite heavily on an interactive command line tool called “UCM”, short for Unison Codebase Manager. New Unison code is typically written in ephemeral “scratch files”, which UCM will automatically watch for changes. Booting up UCM brings us to a blank command prompt. If we create a new file with a .u extension in the current directory, UCM notifies us:

  I loaded /unison/scratch.u and didn't find anything.

Now we can start typing some code. Here’s a simple recursive Fibonacci function:

foo n =
    if n < 2 then n
    else foo (n - 1) + foo (n - 2)

Saving the file, UCM tells us some more info:

I found and typechecked these definitions in /unison/scratch.u.
  If you do an `add` or `update`,
  here's how your codebase would change:
  
    ⍟ These new definitions are ok to `add`:
    
      foo : Nat -> Nat

.>

We can enter update to add this definition to our database of code. Let’s write another function that checks if the n’th Fibonacci number is even. The scratch file now looks like this:

foo n =
    if n < 2 then n
    else foo (n - 1) + foo (n - 2)

fib_is_even n =
    Nat.mod (foo n) 2 == 0

We can call our new functions using watch expressions:

> foo 10
> fib_is_even 10

Putting these lines at the bottom of our file and saving it causes UCM to tell us:

  Now evaluating any watch expressions
  (lines starting with `>`)... Ctrl+C cancels.

    8 | > foo 10
          ⧩
          55
  
    9 | > fib_is_even 10
          ⧩
          false

Let’s add fib_is_even to the codebase as well, as we did for foo. At this point, we can actually delete all the functions from the file with no problems. The watch expressions will keep evaluating correctly since the functions are in the codebase. The scratch file now only contains the watch expressions.

Now for the interesting part. Imagine we later decide to create a new function, also named foo, which does something different than calculating Fibonacci numbers. Let’s make this one append to a string. We’ll change our watch expression with foo accordingly.

foo : Text -> Text
foo n = n ++ ", World"

> foo 10
> fib_is_even 10

Saving the file, UCM informs us:

  If you do an `add` or `update`,
  here's how your codebase would change:
  
    ⍟ These names already exist.
       You can `update` them to your new definition:
    
      foo : Text -> Text

Updating the codebase, nothing breaks! We’ve fundamentally changed the type and name of foo, which fib_is_even depended on, but fib_is_even still functions as intended, since it referenced foo by content, not by name:

    1 | > foo "Hello"
          ⧩
          "Hello, World"
  
    2 | > fib_is_even 10
          ⧩
          false

As you might be able to infer, renaming functions or reusing function names is always safe in Unison, and can never break any code. Neither our own, nor someone else’s. Consequentially, dependency conflicts due to multiple dependencies using the same naming, or due to a dependency updating names or implementations, cannot occur either.

It’s a bit interesting to see how fib_is_even looks now, it actually directly references the hash of the Fibonacci function:

.> view fib_is_even      

  fib_is_even : Nat -> Boolean
  fib_is_even n =
    use Nat ==
    Nat.mod (#b8ohknd8mu n) 2 == 0

We can easily associate a new name with the hash if desired; this is just another constant time operation. It should be evident at this point, that refactoring is quite a breeze in Unison, since it is very difficult to break anything. The Unison codebase can never be in a broken or corrupt state, assuming we don’t manually edit via external means, of course.

In addition to easy refactoring, the hashing mechanism used by Unison has some nice implications for iteration times. In fact, there is no such thing as “building” a Unison codebase. Building (parsing, typechecking) is done immediately when the code is committed to the database and then cached as files on disk. Thus, if I write a recursive Fibonacci implementation once, and commit it to the codebase, I’ll never ever have to build it again, even if I decide to reimplement it with a new name. Furthermore, other people collaborating with me on the same codebase won’t ever have to build it, either. For these reasons, you are almost never waiting around for code to compile while writing Unison. These cached files are even used when adding new code which depends on old code. Since we have the results of typechecking old code cached, for example, Unison can go straight to typechecking just the new code.

Unison doesn’t only just cache compiled code, it also caches test results. To show this, I’ll write a simple unit test verifying our string appending function from earlier:

test> my_test =
    check (foo "Hello" == "Hello, World")

Saving the file, we see:

If you do an `add` or `update`,
here's how your codebase would change:
  
    ⍟ These new definitions are ok to `add`:
    
      my_test : [Result]
  
  Now evaluating any watch expressions
  (lines starting with `>`)... Ctrl+C cancels.

    2 |     check (foo "Hello" == "Hello, World")
    
    ✅ Passed : Proved.

The test passes! If we save the file again, we instead see:

    ...
    ✅ Passed : Proved. (cached)

Since Unison is a purely functional language, and this test doesn’t use any IO, it is provably deterministic. Unison has made use of this, and will never run the test again unless the implementation of foo changes. Instead, we just see the cached test result. You never have to pay the computational price of rerunning tests you didn’t affect with a change, yay!

Due to all of the aforementioned caching, Unison tooling has a pretty nifty set of IDE-like tools. For example, we can trivially search for all functions by type or name, both of which are snappy due to the caching:

.> find : Nat -> Text

  1. base.Nat.toText : Nat -> Text

.> find printLine

  1. base.io.console.printLine : Text ->{IO, Exception} ()
  2. base.io.console.printLine.doc : Doc

I’ve gone on for quite a bit now about hashing and caching, but even ignoring these features, which are where Unison truly innovates, the language is quite pleasant. The tooling is great for such a young language, and it has all the fancy features you’d expect of a modern functional language, like algebraic data types, pattern matching, and parametric polymorphism. It even has a form of effect typing called “abilities” (you can learn more about effect typing in the section on Koka). If any of this piques your interest, definitely try out the language.

APL

Selling point: Terseness taken to the extreme

APL is one of the more well-known languages on this list, but an interesting and influential one nonetheless. The language was initially devised by Kenneth Iverson in the ’60s as an alternative mathematical notation for manipulating arrays, but was turned into a real programming language a few years later, aptly named “A Programming Language”, abbreviated to APL. In other words, APL is old, but manages to feel nothing like other languages of the era.

APL, like Futhark, is in the family of array languages, which, as I mentioned earlier, roughly means that arrays are the primary object of focus, and that the language is built to make operating on arrays convenient and efficient. In fact, APL is often considered the granddaddy of all array languages - a language so influential it spawned an entire paradigm! Among the current generation of programmers, APL is probably most well known for its name, or for being “the language you type with those weird symbols that you need a special keyboard for”. That last part isn’t actually true, by the way. You can type APL on a regular keyboard just fine.

But indeed, APL code consists almost entirely of strange-looking hieroglyphics, making APL code extremely terse. This is one of the main selling points of the language - function implementations in APL are often so short that assigning a name to them seems pointless, as the name itself would be more characters than the function body.

Without further ado, let’s look at some code. I thought it would be fun as a first example to implement the function shown in the Futhark section, which calculates the length of a vector represented as an array. Here is the result:

{(+/⍵*2)*÷2}

Wow… That is pretty short. 12 characters vs Futhark’s 123, to be exact. Despite the length, quite a bit is going on here. Let’s deconstruct it.

In APL, every function takes either 1 or 2 arguments. We call these functions monadic and dyadic respectively (not to be confused with monads from languages like Haskell, which are something different). Because of this, there is no need to name function arguments - they are already named for you. ⍺ is the left function argument, and ⍵ is the right. To define an anonymous function, we use { squiggly brackets } around the function body. Looking at the snippet above, you should now see that we are defining a single argument function, which takes an argument from the right - an array, to be exact.

The first operation we perform on the input array is ⍵*2. * is the power operator, so this code will evaluate to a new array with all elements in ⍵ squared. Next, we use the squared array as the right-hand argument to the reduction operator, /. This operator takes as the left argument a binary operator, and as the right argument an array, and calculates the result of placing the binary operator between each pair of elements in the array. Concretely, +/my_array calculates the sum of elements in an array by interspersing the addition operator +. So, (+/⍵*2) calculates the sum of squares in ⍵. Lastly, we again use the power operator to calculate the square root of this sum, by using ÷2 as the exponent, which is the reciprocal of 2, ie. 0.5.

Phew, that was a mouthful. Let’s try it out in an APL REPL:

    {(+/⍵*2)*÷2}1 2 3
3.741657387

That is indeed the length of the vector [1, 2, 3]. APL array literals are simply space-separated values. To shorten this program further, we could have omitted the anonymous function and placed the argument directly into the body, like so:

(+/1 2 3*2)*÷2

We could also name our function for future use:

    vector_length ← {(+/⍵*2)*÷2}
    vector_length 1 2 3
3.741657387

An interesting property of APL, is that most operators can be used both monadically or dyadically. Take for example the iota operator, ⍳, sometimes known as the index generator. If we apply it monadically to a number n, it gives us an array of the n first numbers:

    ⍳5
1 2 3 4 5

However, when applied dyadically, it gives us the index of the right argument in the left argument. For example, we can use it to find where a number is in an array of numbers:

    8 1 4 3 2⍳3
4

Indeed, 3 is at the 4’th index of the left array.

Another interesting tidbit is that operators (and functions in general) generalize to arrays of arbitrary dimensions. This is known as rank polymorphism. In fact, we’ve already seen this in the first example program, where we used * to exponentiate an entire array. Let’s show a few more examples of this using perhaps the simplest operator of all, addition:

      1 + 5
6

      1 2 3 4 + 5
6 7 8 9

      1 2 3 + 4 5 6
5 7 9

      (1 2 3) (4 5 6) + 1 2
┌─────┬─────┐
│2 3 4│6 7 8│
└─────┴─────┘

Note the use of a 2-dimensional array in the bottom line. Fancy. In APL, anything can be thought of as an array. Even scalar numbers are just 1-element arrays. This property allows the programmer to easily apply the expressive set of operations provided by the language to any kind of data.

The building blocks described so far constitute the core of the language. Quite simple, really. Most of the features I haven’t described are pretty much just syntax sugar. The hard part of learning APL isn’t learning the language semantics, it’s learning all built-in operators and how to effectively chain them together to solve problems. Once you get the hang of it, though, APL can be a very rewarding language to write, as the time it takes to go from an idea for an algorithm to a concrete implementation can be made very short. Due to the terseness of the language, APL also excels in a REPL environment, as a sort of quick calculator on steroids.

As a final piece of piece of shilling, let me show you John Scholes famous implementation of Conway’s Game of Life in APL:

    gen←{({⊃1 ⍵ ∨.∧ 3 4 = +/ +/ 1 0 ¯1 ∘.⊖ 1 0 ¯1 ⌽¨ ⊂⍵}⍣⍵)⍺}
    R∘gen¨ ⍳4
┌─────────────┬─────────────┬─────────────┬─────────────┐
│0 0 0 0 0 0 0│0 0 0 1 0 0 0│0 0 1 1 0 0 0│0 1 0 0 0 0 0│
│0 0 1 1 1 0 0│0 0 1 1 0 0 0│0 0 1 1 1 0 0│0 1 0 0 1 0 0│
│0 0 1 0 0 0 0│0 1 0 0 1 0 0│0 1 0 0 1 0 0│0 1 0 0 1 0 0│
│0 0 1 1 0 0 0│0 0 1 1 0 0 0│0 0 1 1 0 0 0│0 1 0 0 1 0 0│
│0 0 0 0 0 0 0│0 0 0 0 0 0 0│0 0 1 1 0 0 0│0 1 0 0 1 0 0│
└─────────────┴─────────────┴─────────────┴─────────────┘

This shows the first 4 iterations of a 5x7 board initialized to a random pattern. Name another language that lets you write something like this in such a terse manner. I’ll wait.

Prolog

Selling point: Logic programming

Prolog is an interesting language primarily because it is one of relatively few languages in the paradigm known as “logic programming”. It is quite literally in the name, Pro=Programming Log=Logic. In logic programming, the programmer specifies constraints that model some problem domain, and can then execute queries against these constraints to solve problems. Prolog is best explained as a step-by-step tutorial, which is what I intend to do here.

The most basic construct in Prolog is a “fact”, denoted by an identifier, a list of operands, and followed by a dot. In the below snippet, we state several facts. Alice, Jane, Emma, and Sofia are female, while Bob, John, Lucas, and Oliver are male.

female(alice).
female(jane).
female(emma).
female(sofia).
male(bob).
male(john).
male(lucas).
male(oliver).

With this program written, we can start submitting some queries. This is typically done via a REPL. For example, we can ask whether Bob and Alice are male:

?- male(bob).
true

?- male(alice).
false

We can also ask “which people are male” by using an uppercase identifier as the input, indicating an unknown variable.

?- male(X).
X = bob
X = john
X = lucas
X = oliver

The REPL will then print all values of X which make the query true. Facts can also have multiple operands:

parent(alice, john).
parent(bob, john).

parent(jane, emma).
parent(bob, emma).
parent(jane, lucas).
parent(bob, lucas).

We can read this as “Alice is a parent of John”, “Bob is a parent of John”, “Jane is a parent of Emma”, etc. Again, we can make some simple queries:

?- parent(alice, john).
true

?- parent(bob, X).
X = john
X = emma
X = lucas

?- parent(X, john).
X = alice
X = bob

So far, so good. Now for the interesting part. Prolog’s other main construct is the “rule”. Here are some examples of rules:

father(A, B) :- parent(A, B), male(A).
mother(A, B) :- parent(A, B), female(A).

These read as:

“A is the father of B if A is the parent of B, and A is male”
“A is the mother of B if A is the parent of B, and A is female”

Let’s make some more queries:

?- father(bob, john).
true

?- mother(bob, john).
false

?- mother(X, lucas).
X = jane

Rules can depend on other rules, for example, we can express a grandmother constraint:

grandmother(A, B) :- mother(A, C), parent(C, B).

Which reads as “A is the grandmother of B if A is the mother of C, and C is the parent of B”. Let’s see who Oliver’s grandmother is:

grandmother(X, oliver).
X = jane

This is pretty much the core of the language, although the language does have several additional features such as arithmetics and lists. What I find cool about Prolog is that you can model your domain with it, and then use it to “explore” properties of the domain.

Although it might not seem like it from what I’ve shown, general-purpose programming is possible in the language. In some cases, we can even do better than what general-purpose languages provide. As we’ve seen, there are no functions in Prolog, just relations in the form of rules and facts. Therefore, “outputs” are typically just another variable in a relation. To illustrate this, consider the builtin for appending lists:

?- append([1, 2, 3], [4, 5, 6], X).
[1, 2, 3, 4, 5, 6]

Here, X is what we would think of as output in other languages. But since this is just a relation like any other, we can also reverse it and ask “What would we have to append to [1, 2, 3] to get [1, 2, 3, 4, 5, 6]?”:

?- append([1, 2, 3], X, [1, 2, 3, 4, 5, 6]).
X = [4, 5, 6]

We can even ask which lists append together to form [1, 2, 3, 4]:

?- append(X, Y, [1, 2, 3, 4]).
X = [],
Y = [1, 2, 3, 4]

X = [1],
Y = [2, 3, 4]

X = [1, 2],
Y = [3, 4]

X = [1, 2, 3],
Y = [4]

X = [1, 2, 3, 4],
Y = []

How cool is that? The relation works in any direction.

Here’s another example, let’s write some code that can calculate the sum of an array of integers:

sum([], 0).
sum([Head|Tail], Sum) :- sum(Tail, Rest), Sum is Head + Rest.

Readers familiar with functional programming will quickly realize this utilizes recursion. The first line states the fact that the sum of an empty list is 0. The second line is a rule consisting of 2 parts. It states that Sum is the sum of a list consisting of a Head and Tail if Rest is the sum of the Tail, and Head + Rest is equal to Sum. This way of summing a list is nearly identical to how one might do it in a LISP.

We can query it as such:

?- sum([1, 2, 3], X).
X = 6

Unfortunately, unlike the append relation, our sum is not reversible with our current implementation. This is essentially due to limitations of the is operator. There are ways to work around this using more advanced features, namely constraint solvers, but that is out of scope for this post. If this oddball programming paradigm intrigues you, give Prolog a try.

Blogs I love

2022-07-29T00:00:00+00:00

I spend a lot of my time seeking out and reading through technical blogs. I realized today, that my list of go-to blogs only exists mentally. I thought I should write them down somewhere, which is what this post is. I’ll likely be updating this page in the future.

Functional programming

https://bartoszmilewski.com/
Chad category theorist writes about chad category theorist things.
https://amelia.how/
Amazing blog about category theory and functional programming from the main author of 1lab, a category theory knowledge database.
https://las.rs/
A great friend of mine and all around smart guy. Interested in Haskell and Unix in particular.
https://nliu.net/
Haskell blog from biologist turned functional programmer. Some real gems on here.
https://fsharpforfunandprofit.com/
A fantastic blog and learning resource for the F# language.
https://www.haskellforall.com/
Another fantastic Haskell blog from Gabriella Gonzales, author of the Dhall language.
https://plover.com/
Blog from the guy who wrote about monads being burritos.
https://sigkill.dk/blog/
Functional programming and PLT nut. Also my advisor during my thesis work.
https://futhark-lang.org/blog.html
Development blog for the Futhark language, a data-parallel functional language targeting GPUs.
http://blog.vmchale.com/
Array languages and functional programming galore.

Unity stuff

https://bgolus.medium.com/
The KING of the Unity forums.
http://jbooth.blogspot.com/
The KING of the Unity asset store.
https://blog.orels.sh/
Friend of mine. I like what they’ve written so far. Mostly about Unity-specific shader stuff.

General tech

https://mtlynch.io/
General tech blog I found recently. Writes a lot about entrepreneurship.
https://overreacted.io/
General tech, loosely JavaScript focus blog. Articles read nicely, and have great design.
https://blog.codinghorror.com/
The famous codinghorror blog. Everyone should know this one.
https://acko.net/
Great, long-form, in-depth articles about everything from GPU programming to pure mathematics. Also, the website design is impeccable.
https://drewdevault.com/
Don’t know this one too well yet, but read a few articles and found them interesting.
http://cnlohr.blogspot.com/
Friend of mine and certified C-boomer. Doesn’t post very often.
https://boxcat.site/
Only contains one post so far about speeding up the VRChat Udon virtual machine. It’s neat.

Techno wizardry

https://wunkolo.github.io/
SIMD and graphics chad. Writes great articles about some seriously specific technical stuff. Juicy details.
https://blog.demofox.org/
The blog of Alan Wolfe. Another one every programmer should know. Writes about a large range of complex technical topics, with great attention to detail.
https://justine.lol/
Lots of super low-level juice details and technofuckery.
https://gankra.github.io/blah/
Technical crimes committed by the author of the Rustonomicon.
https://fasterthanli.me/
Very long form exploratory style articles, mostly about Rust.
https://aphyr.com/posts
Person behind the great “(adjective) the coding interview” series of blog posts. Funny stuff.
https://mort.coffee/
Don’t know too much about this guy, but they mostly write about C.
http://paulbourke.net/
Website of the legendary Paul Bourke. Not really a blog, but great technical pages and documentation on stuff you’ve probably never heard about.
https://psychopath.io/
Blog about a path tracer. Lots of great in-depth articles on here.
http://timjones.io/
Shader and graphics guru.

Shaderception - a deep dive

2022-04-28T00:00:00+00:00

Intro

A while back, I wrote a compiler for a shading language, named Shaderception. The compiler runs within the social VR platform VRChat, which poses some interesting and odd technical challenges. The result was a virtual world accessible through VRChat, which functions similar to Shadertoy and lets users write and execute interactive shaders entirely from within the platform. The name comes from the fact that the runtime for the language is, itself, written in a fragment shader.

I’ve briefly written about the project in the past, but have been meaning to write a more in-depth technical summary of the techniques I employed to make it all happen. The project consists of several distinct parts, each of which I’ll describe separately. As a goal, I’ll try to keep it fairly understandable even to people who don’t know much about compilers. Apologies if I come off as condescending at times.

The language

The language compiled by Shaderception, dubbed Shaderception language (I’m amazing at naming things), is fairly similar to HLSL with a few major differences. To motivate the rest of the post and show what we are working towards, here is an example program that calculates an image of the mandelbrot fractal:

fun squareComplex(z) {
	float2(
		z.x * z.x - z.y * z.y,
		z.x * z.y + z.y * z.x
	)
}
fun squareLength(a) {
	a.x * a.x + a.y * a.y
}
let maxSteps = 20;
let p = float2(-2.5 + (1.0 - (-2.5)) * uv().x,
               -1.7 + (1.7 - (-1.7)) * uv().y);
let z = p / (mod(time().y, 20.0) / 3.0);
let steps = 0;
let i = 0;
while (i < maxSteps) {
    if (squareLength(z) < 4.0) { 
        z = squareComplex(z) + p;
        steps++;
    }
    i++;
}
if (steps == maxSteps) {
    float4(1.0, 0.0, 0.0, 1.0)
} else {
    float4(steps / 15.0, 0.0, 0.0, steps / 15.0)
}

Which outputs this image:

Those familiar with HLSL will immediately notice a few major differences. Roughly, they are:

Everything in Shaderception lives in a single global scope, where both functions and raw code are permitted. The last expression in the program will be the color of the pixel being shaded.
The language is dynamically typed, simplifying the syntax for defining functions and variables quite a bit.
The language uses implicit returns, just like Rust. The last expression in a function without a semicolon after it will be returned.
The let keyword is used to declare or reassign variables. It is actually completely optional, since every variable is global.

Some other notable points which aren’t immediately obvious from the snippet:

There is a maximum of 256 variable names per program, which is simply a technical limitation.
Recursion, and in general reentrant functions are not allowed, which is similar to HLSL.
There are no matrices, only scalars and vectors of dimension 2-4.
No switch statements.

The language was explicitly designed to be easy to compile with a stupid simple compiler, and to be executable with a stupid simple runtime, while maintaining a similar feel to existing shading languages where it was practical.

The compiler

VRChat has 2 main ways of running user code: Shaders and Udon. It’s completely unreasonable to attempt to write a compiler in a shader, so the compiler itself uses Udon. Udon is a scripting system which consists of a simple instruction set, usually referred to as Udon assembly, and an accompanying node-based visual scripting tool. For people who prefer to write code instead of using node graphs, we have Merlin’s fantastic UdonSharp compiler, which compiles a subset of Unity-flavored C# to Udon assembly. This is what I utilized to create the compiler.

On UdonSharp

UdonSharp is similar in spirit to normal C#, but there are some very notable differences that make the former a very strange environment to write larger programs in. Among these differences are:

The speed. UdonSharp code is extremely slow, usually several thousand times slower than the equivalent C#. This is no fault of the UdonSharp compiler - the underlying scripting system is just slow as a consequence of its design. What slows Udon are calls to external .NET methods, which have large overhead. This is such a big deal that asymptotic complexity becomes largely irrelevant, and the only factor that really matters for optimization is minimizing the amount of extern calls.
There are no generics, which means no List<T> or Dictionary<K, V>.
Splitting a program into multiple files is tedious, as each UdonSharp program must map to a behavior, which forces some restrictions on how the class is written. Inheritance becomes less powerful, and there is no simple way to define a quick plain-old-data struct or class for use within the program. Also, each behavior brings overhead.
Many types and methods existing in regular .NET or in Unity’s APIs are not exposed to the Udon scripting system. This is done intentionally to sandbox user scripts, but often means that the idiomatic .NET solution to a problem simply isn’t possible.

For these reasons, the compiler is written as a single 1300 line C# script, containing a single class, and which uses arrays as the sole data structure, primarily arrays of object or string. Data structures such as hashtables are implemented via linear search through arrays storing keys and values in an alternating fashion, since real hashtables simply weren’t worth the overhead in extern calls. An amazing code style, I know.

Immediate mode compilation

Writing compilers is difficult work. Writing compilers in such a limited programming environment is not only difficult, but extremely tedious. To prevent myself from going insane and make it all easier to implement, I made a few fairly unorthodox design decisions. The main one being a property of the compilation pipeline which I shall refer to as immediate mode compilation (if you have a better word for it, please let me know, I’m not aware of one).

To understand what that is, let’s talk briefly about architecting a compiler. If you’ve ever taken a course on the subject, you were probably taught that most compilers should consist roughly of the following distinct ‘parts’ aka. pipeline stages:

Lexing/Tokenization
Parsing
Typechecking and semantic analysis
Optimization passes
Code generation

Each of these being like a separate little program that takes input from the previous part and passes it to the next. Source code is first lexed to produce a stream of tokens, which is parsed to construct an abstract syntax tree (AST). This tree structure is then used to check proper usage of types and program semantics, and later optimized into a semantically equivalent, but more performant form. Lastly, the syntax tree, or some other intermediate representation, is used to generate code in the target language (such as assembly).

Implementing all of this in the aforementioned limited environment is quite ambitious, so I started cutting stuff away. If I make the language dynamically typed, there is no need for type checking, which removes one pipeline stage. Another pipeline stage is removed by the decision to not make an optimizing compiler - if you write slow code, you will pay the price. Also, due to the lack of proper user types, tokens produced by the lexing stage are actually just objects, that can be either strings in the case of identifiers or keywords, floats in the case of literals, or chars in the case of symbols like + or ) - a disgusting use of polymorphism. All of this leaves us with the following pipeline:

Lexing to objects
Parsing
Code generation

Even this pipeline, however, I considered too ambitious. It turns out representing and working on more complex data structures, such as an abstract syntax tree, is pretty difficult and unwieldy with Udon due to the lack of proper user types. To circumvent dealing with that headache at all, parsing and code generation are done in one single step, which is what I mean by ‘immediate mode compilation’. Code in the target language is immediately generated while the token stream is being read.

To give an example, the code to compile a while loop looks like so:

[RecursiveMethod]
void WhileLoop()
{
    string startLabel = "start_" + labelCount++;
    string endLabel = "end_" + labelCount++;

    Eat("while");
    Eat('(');
    
    Emit("LABEL", startLabel);
    Expression(); // condition
    Emit("CONDJUMP", endLabel);

    Eat(')');
    Eat('{');

    Block();
    Emit("JUMP", startLabel);
    
    Eat('}');

    Emit("LABEL", endLabel);
}

There are 2 main operations being performed in this code: Eat and Emit. Eat takes as input a token we expect to be present, consumes the next token in the token stream, and produces an error if it wasn’t what we expected. Emit produces an instruction in the language used by the compiler as a compilation target, which I’ll describe in further detail later. If you’ve ever written an assembly language, you may recognize some of these instructions. Expression and Block are recursive functions which look similar to the one above, and which compile a single expression and a block of code respectively.

Thus, the code functions roughly like so:

Eat a while keyword token.
Eat an opening parenthesis (.
Place a label in the generated code, which is just an indicator of a position in said code that we can jump to later. This one indicates the start of the loop body, including the loop condition.
Compile an expression - the loop condition we want to test.
Place a conditional jump CONDJUMP into the generated code, which will jump to the end of the loop if the loop condition evaluates to false.
Eat a closing parenthesis )
Eat a opening bracket {, indicating the start of a code block.
Compile a bunch of statements that make up the loop body.
Place a JUMP instruction in the generated code which jumps back to the start of the loop and tests the loop condition again.
Eat a closing bracket }.
Place another label in the generated code, this time indicating the end of the loop.

Phew, that was a bit of a mouthful, but I hope you see how this mixed approach cuts down on the amount of code I have to write quite drastically, at the cost of robustness and scalability; certains language constructs are very tricky to compile with this approach, and it implicitly puts limitations on the kind of languages we can make, since we don’t really have a way to ‘look ahead’ and make decisions about how to compile something. For this same reason, the syntax was intentionally chosen to be fairly unambiguous.

Aggresive inlining

Most languages have a notion of a callstack, which is a stack data structure that stores several pieces of data vital to supporting function calls. Usually, that includes an address to return to after a function is finished executing, and the values of local variables used within the function. For general purpose languages, the existence of a callstack is essentially assumed, since it is so common. However, this infrastructure has a complexity toll on both the compiler and target instruction set - both things I was trying to make as simple as possible. Therefore, Shaderception does not support function calls.

Wait, what? No function calls? But you just showed a piece of code in the start of this post that defined and called several functions!

Indeed, Shaderception does support functions, both user defined and builtin, but has no proper function calls. Instead, every single time we encounter a function call, we inline the function at the callsite. For those unfamiliar with the term, this literally means to remove the function call and replace it with a copy-pasted version of the body of the function being called. This process is necessarily recursive, since those functions may call other functions.

Here as an example of how this works at a semantic level:

fun bar(x) { x + 1337 }
fun foo(y) { bar(y) + 420 }
let baz = foo(69);

Would turn into:

fun bar(x) { x + 1337 }
let baz = bar(69) + 420;

And then finally into:

let baz = (69 + 1337) + 420;

As is evident from this example, all function calls have been removed, yay! And if there are no function calls, there is no need for a callstack or any other kind of data structure to keep track of calls. In reality, functions aren’t inlined at the source level, but instead after the code is compiled.

This does, however, come at a cost. Inlining massively bloats the size of the generated code. If you define a function and call it 10 times, suddenly the body of that function will appear 10 times in the generated code! This can be quite problematic for larger programs.

Additionally, recursion and reentrancy becomes impossible, for example, if we tried to inline the following function:

fun foo(x) { foo(x) + 1 }

What should it become? The inlining process would never terminate, since this can be inlined an infinite amount of times, and there will still be a function call present. This is the reason recursion is forbidden in the language. This isn’t actually so insane - most shading languages don’t support recursion for very similar reasons.

Extern calls

No language is complete without a way to perform input and output. In shaderception, this is facilitated with a bunch of builtin, external functions. This includes stuff like uv() to get the current pixel coordinates, but the same mechanism is also used for the typical expected math functions like sin, round, log etc. These functions are not written in Shaderception language, or even in the instruction set targeted by the compiler. Instead, they are written directly in HLSL, hardcoded into the runtime. An instruction CALL exists to call these external functions, and takes as input an integer identifier associated with each builtin. Notably, the CALL instruction only works with builtin functions - it isn’t possible to define your own that can be called with this mechanism.

Label linking and register allocation

When describing the various pipeline stages of the compiler I made a slight distortion of the truth. In fact, more work is done after the initial round of code generation, namely label linking and register allocation.

Note: Both of these are slight misuse of technical terms, they are just simply names I’ve chosen to describe what is being done.

While generating code, we have been placing labels throughout that were used as the target of various jump instructions. But how do we know which memory address to jump to concretely when all we have is the name of the label? Label linking addresses this, by traversing the generated code, and keeping track of the offsets of labels in the generated program binary as it goes. Every time a jump instruction is encountered, the target of the jump is replaced with the concrete offset in the binary. Label instructions are also removed in this process

Given a (pseudo) program binary that looks roughly like so:

JUMP BAR
LABEL FOO
SETVAR BAZ
LABEL BAR
PUSHCONST 1337
JUMP FOO

We would transform it to this:

JUMP 3
SETVAR 42
PUSHCONST 1337
JUMP 2

When that process is done, we perform register allocation. This solves a similar problem as we had with labels, but for variables. Another consequence of not having a callstack is that there isn’t an immediately obvious place to store local variables. To solve this in the simplest way possible, all variables are stored in a globally accessible variable store, which can hold at most 256 variables, which one may also call registers. Register allocation is the process of mapping variable names to positions in that variable store.

Note: Again, this is slightly different to what is usually understood in literature by register allocation, but similar enough in spirit that I have no shame in reusing the term.

Given a program binary like so:

PUSHCONST 1337
SETVAR FOO
PUSHCONST 42
SETVAR FOO
PUSHCONST 69
SETVAR BAR

We would transform it to:

PUSHCONST 1337
SETVAR 0
PUSHCONST 42
SETVAR 0
PUSHCONST 69
SETVAR 1

Assigning the variables FOO and BAR to position 0 and 1 in the variable store respectively. If you go over 256 variables, you get undefined behavior. Everybody loves undefined behavior. In a real compiler, going over the max register count would result in a ‘spill’, in which spilled variables are stored in memory instead of a register.

Producing a binary

All the program binaries and generated code I have shown so far have been in mnemonic form. The very final step of the compilation process is to convert this into a more practical, binary format. Of course, we don’t actually store the string "JUMP" for every jump instruction, that would be insane. Thus, every instruction maps to a distinct operation code, or opcode for short. This is a straight forward matter of traversing the generated code in mnemonic forms and putting opcodes into a list which becomes the program binary.

The virtual machine

So far, I’ve only described the process of compiling a source program to a binary format. Without a machine to run that binary, the compiler is largely useless. Most languages use one of three approaches to make the programs their compilers produce executable:

Target an instruction set native to the computer you wish to run the program on, in which case you are done at this step.
Interpret and execute the source program or a simplified version of it directly. This is quite flexible but can also be horribly slow and implies writing code to execute something much more complex than a program binary, which isn’t ideal when trying to minimize complexity.
Target an imaginary instruction set and write a program that pretends to be a machine that can execute this instruction set - a virtual machine. This kind of instruction set is usually referred to as ‘bytecode’. This is essentially a hybrid of the 2 other approaches, and inherits some nice benefits from both.

Frankly, the third and last option was the only viable one for this project, since the runtime (concretely, the virtual machine) will be implemented in a fragment shader. Now, why the hell would anyone write a virtual machine for a language like this in a shader? Shader languages are not general purpose and typically not well suited for programs like this.

Actually, I gave the answer to this near the start of the post. Udon is slow. Very slow. Extremely slow, even. Trying to write a virtual machine to execute our program in Udon is a complete lost cause - we would barely be able compute the color of a single pixel without lagging everyone out. GPUs on the other hand are very fast, and we can leverage this to make the language at least somewhat performant. Usually, this too is a bad idea, since general purpose languages cannot easily make use of the parallel nature that GPUs are famous for. Shaderception language is not general purpose though, and it turns out writing a virtual machine in a shader, for a shader language spefically, isn’t actually such a terrible idea. The kind of programs we will become are parallel in nature, as all fragment shaders are, which will map nicely to the parallelism of the host GPU.

Stack or register based?

In the world of virtual machines (and actually real machines, as well), there are 2 main approaches: Stack or register based. A good example of register based instruction sets are x86 and ARM. Instructions for register machines refer directly to registers on said machine, of which there is usually a fairly limited amount. If we wanted to add 2 numbers and assign it to a variable in something similar to x86, for example, it may look something like this:

MOV   %eax, 1337
MOV   %ebx, 420
ADD   %eax, %ebx 

Where %eax and %ebx are two registers, and we are assuming the variable we are assigning to is being stored in %eax. MOV moves a value from one location to another, and ADD adds a value to another value stored at a given location.

In a fictitious stack based instruction set, the equivalent may look like so:

PUSHCONST 1337
PUSHCONST 420
ADD
SETVAR    res

Stack machines use an ever-present stack to perform their operations. Each instruction may push and pop values on the stack to perform calculations. In this example snippet, we first push the values 1337 and 420 onto the stack. Next, we execute the ADD instruction, which will pop the 2 topmost values of the stack, add them together, and push the result onto the stack. After this operation, the stack will contain 1 value, the number 1757. Lastly, we execute SETVAR which will pop a value from the stack and assign it to a variable. res is assumed to be some identifier referring to the variable we care about.

Note: This ‘stack’ shouldn’t be conflated with a callstack. Instead, you may perhaps call it a value-stack. This stack is used constantly, for almost every operation, unlike the callstack, which is only really used when calling functions. That being said, this stack may be used to store return addresses and local variables in a more sophisticated language.

For Shaderception, I designed a stack based virtual machine. The reason for this choice is very simple: They are dead simple to implement. A reoccuring theme in this project.

The Shaderception instruction set

Before I describe the instruction set I decided on, a few things are worth noting. Every instruction consists of 2 float4s: An opcode and operand (wasteful I know, bite me). The opcode, which is an integer, goes in the X channel of the first float4.

Without further ado, here is the instruction set specification used by the Shaderception virtual machine, and generated by the compiler:

`1 - PUSHCONST (float4)`

Push a constant floating point vector onto the stack

`2 - PUSHVAR (id)`

Push a variable stored at location id to the stack

`3 - BINOP (op)`

Take the 2 topmost elements of the stack, perform binary operation.

`4 - UNOP (op)`

Take the topmost stack element and perform unary operation on it, put result on stack.

`5 - CALL (id)`

Call builtin function with the given ID. Parameters should be on the stack when the instruction is invoked. Result is put on the stack.

`6 - SETVAR (id)`

Pop a value off the stack and set the variable at location (id) to the value. This operator also stores a mask of which elements of the vector to set in the y element of the opcode. An example of such a mask could be the float “1223.0”

`7 - JUMP (loc)`

Jump to a specified memory location.

`8 - CONDJUMP (loc)`

Pop a value off the stack. If it is false (equals 0), jump to the specified location.

Executing the instructions

The core of the bytecode virtual machine is actually really simple. It is essentially just a loop that runs until the program terminates, and which contains a giant switch statement that switches over the opcode of the next instruction in the program binary. The start of it looks like so:

for (uint i = 0; i < 4092; i += 2)
{
    uint instr = round(_Program[i].x); // opcode
    float4 opf = _Program[i + 1];      // operand
    uint opi = round(opf).x;           // first part of operand
    [forcecase] switch(instr)
    {
        case 1: // PUSHCONST <float>
            pushStack(opf);
            break;
        case 2: // PUSHVAR <char>
            pushStack(getVar(opi));
            break;
    // and more down here ...

To store the variables and the stack, I use 2 little arrays and an offset pointing to the top of the stack:

static float4 stack[128];
static uint stackPtr = 0;
static float4 vtab[256];

Similarly, the program binary is stored in a large array, and the instruction pointer is simply the loop index in the previous loop.

External calls are the only real non-trivial instruction. For these, I first use the integer ID of the extern to determine its arity, then pop that many values off the stack, and finally call the extern using those popped values, putting the result back onto the stack. ‘Calling the extern’ in practice means using a horrendous giant switch statement that switches over the ID and contains the implementation of every single extern. A similarly gross switch statement is used to determine the arity.

The start of that function looks like so:

float4 callFun(uint opi, float4x4 ops, uint4 dims)
{
    [forcecase] switch(opi)
    {
        case 1:  return log(ops[0]);
        case 2:  return log2(ops[0]);
        case 3:  return sin(ops[0]);

Quite simple, really.

Dynamic typing is cursed

Shaderception language is dynamically typed. In hindsight, this was both a good and a bad decision. The good was that it massively simplified the compiler. The bad was that it made the virtual machine more complex, and complex shader code kind of has a tendency to fail for seemingly random, mysterious reasons. In addition to just being dynamically typed, further complexity is added by implicit conversions between vector types being supported, and by builtin functions being polymorphic.

Every single value in the Shaderception is stored as float4. In order to determine the width of a vector, unused channels are set to NaN (this is a horrible idea for the love of god don’t do this). Some examples:

float4(1337.0, NaN, NaN, NaN) // This is a scalar
float4(13.0, 23.0, 34.0, NaN) // This is a 3D vector
float4(1.0, NaN, 3.0, NaN)    // This is invalid

By counting the amount of NaNs in a vector (done simply with a dot product), I can thus get the dimension of the vector, and base behavior on it. A good example of why this is necessary is the following snippet of Shaderception language:

let a = 3.0;
let b = float3(1.0, 2.0, 3.0);
let c = dot(a, b);

This is completely valid, and what should happen is that the value ‘a’ is implicitly cast from a scalar float to a vector float3, and then we take the dot product of (3.0, 3.0, 3.0) and (1.0, 2.0, 3.0). If we did something naive like, say, directly calling dot with the 2 operands and without the NaN encoding, we would take the dot product of (3.0, ?, ?) and (1.0, 2.0, 3.0) where ? is a placeholder for whatever values happened to be in the channels that we didn’t care about, since ‘a’ was a scalar. This can lead to some seriously wrong results. Certain functions may also behave differently depending on what type of value was passed, which this approach addresses.

NaN propagation is cursed

Play stupid games, win stupid prizes. This is the thought that should immediately have sprung into my mind when I came up with the idea of using NaN to encode types. For whatever reason, I did it anyways. One thing I learned pretty quickly is that NaN does not care about your feelings, and NaNs may propagate in very strange ways in HLSL.

For example, dynamically indexing a vector that contains a NaN in one channel like so:

// assume foo is a vector (0.0, NaN, 0.0, 0.0)
float bar(uint i) {
  // this returns NaN regardless of 'i'
  return foo[i];
}

I wrote an entire post about similar issues, so won’t go into further detail.

How not to crash your GPU driver

The initial implementation of the virtual machine caused quite a few graphics driver crashes. I did 2 things 2 mitigate this.

The first was a jump counter that is incremented every time a jump instruction is performed. If this ever reaches some arbitrarily set maximum value, the program simply terminates. This prevents programs that manage to loop infinitely from causing timeouts and crashes.

The second was AMD specific. The small arrays shown in the previous section used to store variables and values on the stack would cause spurious crashes on several AMD cards. The fix turned out to be to always limit the passed index to the range of the array when indexing the array, like so:

float4 getVar(uint opi)
{
    return vtab[opi % 256];
}

My code never actually did any read or writes out of bounds, but apparently the mere thought of this happening was enough to cause some AMD drivers to shit themselves.

The tooling

In addition to the compiler and the virtual machine, the Shaderception project consists of a few, additional, projects of varying interest.

The Shaderception world

The Shaderception world, which you can check out on VRChat here, even if you don’t have a VR headset. Is the main way to interface with Shaderception. It consists of a fancy text input box, and a few screens that show the output. The world also has video players and cameras that feed their data to the Shaderception system, such that you can play around with them and make image-based effects.

Alternatively, here is a tweet that shows it off.

User submitted programs

I’d like to thank my good friends who have submitted their own programs written in my botched together language to me.

By far the most notable is by my great friend Fuopy, who wrote an entire playable Pong game played against a simple AI (depicted above). Not only is this quite a feat in domain-specific shading language, but I’m particularly impressed by how he managed to get this done while I was still working on the compiler and fixing bugs along the way. Thanks for the great motivation, Fuopy. He posted a tweet about it here.

Another person who contributed some very neat programs was M.O.O.N. They wrote some trippy image-based effects, and ported some neat shaders they had written previously to the language. Thanks for that.

Shaderception cross

Shaderception is cool, but HLSL is also cool. Therefore, a compiler that takes Shaderception code and produces HLSL is per definition doubly cool. This is exactly what shaderception-cross is - an entirely separate, second compiler for Shaderception that emits HLSL.

Why do I do this to my self.

One major challenge in writing this was the types. Since I am going from an untyped to a typed language, all the types have to be inferred from their use. This resulted in quite some spaghetti code for the type inference algorithm, but the source code is in general much nicer to read. If you want to look at a Shaderception compiler but also not feel dead inside while you do so, I suggest looking at this one.

Shaderception standalone

I’ve written stubs for most of the Unity-only functionality I’ve used in the compiler. This allows Shaderception to be run outside of VRChat, and more generally outside of Unity. This was useful when testing out the compiler without having to spin up Unity each time. The link to this project is here.

Shaderception VSCode syntax

I’ve written a tiny plugin for VSCode that adds syntax highlighting for Shaderception language. The source code is available here, and the plugin on the marketplace here.

Weird HLSL tricks Unity doesn't want you to know about

2022-04-17T00:00:00+00:00

Intro

A while back, I wrote a compiler for a shading language that targets a stack based virtual machine which is, itself, written in an (Unity-flavored) HLSL shader. The project was aptly named Shaderception, and was what I consider to be a true display of esotericism - a trait I value highly. While writing it, I learned a few nifty and/or evil tricks, which I’ve been meaning to write down for some time. Let’s get to it.

Dynamically assigning to arrays

If you have ever written HLSL code that makes use arrays, you may have run into issues when trying to assigning values to non-constant indices in the array. For example, the following snippet causes a compilation error:

uint _Idx;
float _MyArray[100];

void bar() {
    // ERROR: array reference cannot be used as an l-value;
    // not natively addressable
    _MyArray[_Idx] = 20;
}

The default qualifier for a variable in global scope in HLSL is uniform, and it turns out code like this is actually legal, just only on non-uniform arrays. Thus, you can prevent the error by adding the static qualifier:

uint _Idx;
static float _MyArray[100];

void bar() {
    _MyArray[_Idx] = 20; // No error :D
}

This does, however, mean that you can no longer assign the array externally - it exists only within the shader code itself. This can still be very useful, though. I use such arrays to store variables and values on the stack in the aforementioned virtual machine:

static float4 stack[128];
static uint stackPtr = 0;
static float4 vtab[256];

It is possible to index in this way into uniform arrays, but doing so requires a redundant loop:

uint _Idx;
float _MyArray[100];

void bar() {
    for (int i = 0; i < 100; i++) {
        // No errors here :P
        if (i == _Idx) _MyArray[_Idx] = 20;
    }
}

Dynamically assigning to vectors and matrices

You can assign to elements of a vector or matrix dynamically too, but sadly, it requires a redundant loop again. Here is an example:

uint _Idx;

void bar() {
    float4 myVec = 0;
    float4x4 myMat = 0;
    for (uint i = 0; i < 4; i++) {
        if (_Idx == i) {
            myVec[_Idx] = 1337;
            myMat[_Idx] = myVec;
        }
    }
}

NaNs are weird

The shading language I wrote supports scalars as well as vectors of different length, just as HLSL. When writing the virtual machine, I had the horrible idea of encoding the size of vector directly in each value by using NaNs to indicate unused channels, and store everying as a float4. For example, a scalar would be encoded as float4(value, NaN, NaN, NaN) and 3D vectors would be float4(value1, value2, value3, NaN) etc. This allowed me to handle typechecking and implicit casting directly in the shader, but at the cost of several painful hours of debugging seemingly nonsensical output. It turns out the HLSL really does not care about your feeling whens NaNs are involved. Here are some key points:

`isnan()` is a lie

HLSL has an intrinsic for detecting NaN values, isnan(x). Unfortunately, in Unity’s setup, it will lie to you. For example, isnan(sqrt(-1)) evaluates to false. This is because the compiler aggresively optimizes away NaN checks. A workaround is XOR the input with an unbound uniform (which in Unity will be initialized to 0 by default), forcing the compiler to be unable to optimize. Here is a function that you can use instead of isnan:

uint _Stupid;
float isActuallyNan(float val) {
    return isnan(asfloat(_Stupid ^ asuint(val)));
}

Producing NaNs

One must be careful when intentionally producing NaN values. Methods such as sqrt(-1) or (0.0/0.0) do work, but the compiler may erroneously compile out the NaN while doing optimization, such as constant folding. Unbound uniforms to the rescue again - you can use sqrt((-1)^_Stupid). Additionally, those 2 aforementioned methods will generate an annoying warning, so I actually use asfloat(-1) instead, which causes no warning.

Dynamic indexing causes NaN propagation

When accessing an element of a vector or matrix using a dynamic index, if the right hand side of the assignment contains a NaN, the NaN will spread to every element! Consider the following snippet

uint _Idx;

bool foo() {
    float4 oneNan = float4(1, sqrt(-1), 1, 1);
    return isActuallyNan(oneNan[_Idx]);
}

This will return true, even when _Idx is explicitly set to 0. The issue occurs in this expression oneNan[_Idx] which will evaluate to a vector containing only NaN! And the exact same happens for matrices as well.

`[forcecase]` is great

Writing a virtual machine usually involves some large switch statements. Not too many people seem to know that switch statements, as with other control flow, can be annotated with compiler directives dictating how they are compiled. For switch statements, they are listed on MSDN. The default in most cases will be a series of if-statements, corresponding to the [branch] directive. In almost all cases, though, [forcecase] will yield much better performance, especially when writing cursed abuse of shader code like I was doing for this project. As far as I understand, this basically forces the generation of a jump table. The only caveat is that the compiler will occasionally refuse to compile code using [forcecase]. :(

Constant buffer aliasing

Unity has a limitation on the length of uniform arrays, 1023 to be precise. This limitation comes from limitations of constant buffers, which are used implicitly. In HLSL, a constant buffer can hold 4096 vectors, each with four 32 bit values. Since arrays can also contain matrices, each of which contain 4 vectors, the max amount of 4x4 matrices in a single constant buffer is 1024. The 4x4 matrix is HLSL’s largest primitive type, and is what Unity’s limitation is based on. I’m not sure why the limit is 1023 and not 1024, though. You can work around this limitation using a trick I shall refer to as “cbuffer aliasing”. Here is some code demonstrating it:

cbuffer MyCBuffer {
    float4 _MyArray[1023*4] : packoffset(c0);  
    float4 _MyArray0[1023] : packoffset(c0);
    float4 _MyArray1[1023] : packoffset(c1023);
    float4 _MyArray2[1023] : packoffset(c2046);
    float4 _MyArray3[1023] : packoffset(c3069);
};

By adding 4 unused arrays and manually specifying their packing offsets in the constant buffer such that they overlap with a single, larger array, we can effectively get a much larger array, one that actually reaches the limit of data in a single constant buffer. With this, we can use _MyArray in our code however we want. Though one more thing needs to be done for this to work. If we don’t ever use _Program0 through _Program3, the compiler will optimize them away. Crucially, the compiler must believe that they contribute something to the final output of the shader. My usual trick is therefore to use them in a calculation that I know will never will be hit:

// Hack to prevent Unity from deleting aliased cbuffer
if (uv.x < 0) output = _Program0[0] + _Program1[0] + _Program2[0] + _Program3[0];

...

return output;

Here, uv is input UV coordinates to the fragment shader.

The end

That was all for now. You can find more tricks at my shader-knowledge repo.

Setting up Haskell Language Server with VSCode is cursed

2022-04-06T00:00:00+00:00

Haskell setup is pain

I’ve been enjoying writing Haskell a lot lately, but my god has the tooling been a mixed experience. I recently bought a new laptop, which I’ve installed Arch Linux on, and wanted to set up the Haskell language server (HLS) to work with Visual Studio Code on it. Every time I’ve attempted this in the past, it has been extraordinarily painful. Here is a cheatsheet to my future self as well as anyone who may find it useful.

Template Haskell

The project I am working on uses template Haskell fairly heavily. Unfortunately, most easy ways to install HLS default to static linking, which doesn’t play well with with template Haskell. In fact, it plays so badly that the language server will routinely segfault on my project on several machines I’ve tried to do this on.

GHCUP to the rescue

By far the least painful way I have found to get this all working is using ghcup. I recommend installing every piece of tooling needed with it (ghc, cabal, hls, stack).

Installing GHCUP

If you have Haskell installed in any other way, uninstall it, and follow the steps on https://www.haskell.org/ghcup/. Concretely, run:

curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | sh

You will prompted to install a few different tools. Only install ghc, cabal, and stack, but decline to install the language server. Make to sure to allow the tool to add the tools to your $PATH.

If your project needs as specific version of ghc, you can easily install and switch to it using the ghcup tui interface. My project used GHC 9.0.2.

Installing HLS

For the language server, we need to build it from source to support template haskell properly. There are some steps here https://haskell-language-server.readthedocs.io/en/latest/troubleshooting.html#static-binaries. Since we are using GHCUP, run something like:

ghcup compile hls -v 1.6.1.0 --ghc 8.10.7

With your appropriate version. Some versions apparently need a specific cabal.project, so I ran:

ghcup compile hls -v 1.6.1.0 --ghc 9.0.2 --cabal-project cabal-ghc90.project

That will compile HLS, which might take a little while.

Making VSCode aware of binaries from GHCUP

GHCUP will place a line into your .bashrc that looks like so:

[ -f "/home/pema99/.ghcup/env" ] && source "/home/pema99/.ghcup/env" # ghcup-env

Which works fine when using the installed binaries via a shell. Unfortunately, under normal circumstances, the $PATH VSCode uses will not be affected by this. (More info here https://github.com/haskell/haskell-language-server/issues/236). We have a few options to fix that:

Add this same line from above, to the bottom of your /etc/profile file. This is a bit of a hack, but it is what I did.
Start VSCode from the terminal with the flag --force-user-env. Super tedious.
Use the haskell.serverEnvironment setting in your VSCode settings.json to manually make the Haskell extension aware of where the binaries installed by GHCUP are (info here https://github.com/haskell/haskell-language-server/issues/236#issuecomment-987174473)

Setting up the Haskell extension

For whatever reason, the VSCode Haskell extension doesn’t seem to play nicely with projects that have never been built, so I suggest doing the following:

If you don’t already have the Haskell extension installed, open a fresh VSCode instance and install it from the marketplace. https://marketplace.visualstudio.com/items?itemName=haskell.haskell
Optionally make the edit to your settings.json if you opted for that in the previous step.
Close VSCode
Navigate to your Haskell project in a shell
Assuming you are using stack to build, do stack setup then stack build.
Now, open VSCode in your project directory.

The end

If you followed every step correctly, VSCode should now automatically figure out your setup, start compiling your code, and hook up the language server. Or maybe not, if I made some stupid blunder while writing this. Let’s hope not, future me will surely be annoyed.

What the hell is a monoid

2022-03-03T00:00:00+00:00

For some reason, I keep getting asked the same few questions about common constructs in functional programming. I want a page to point friends towards. That is what this is. Without further ado, welcome to the 999th terrible monad tutorial.

What is a monoid

A monoid is a set equipped with an associative binary operation and a neutral element.

For those who may not remember, the associative property holds for an operator ‘$\oplus$’ if it is the case that $(a \oplus b) \oplus c = a \oplus (b \oplus c)$.

A neutral element is an element ‘$z$’ of the set such that $a \oplus z = z \oplus a = a$.

Here are some examples of monoids, specified as (the set, the operator, the neutral element):

(integers, +, 0)
(integers, *, 1)
(strings, string concatenation, the empty string)
(lists, list concatenation, the empty list)
(functions, function composition, identity function)

What is a functor

A functor is, essentially, a data structure containing values of some generic type, and which can be mapped over. In Haskell, the function to perform the mapping is called fmap and has type

Functor f => (a -> b) -> f a -> f b

in Haskell. In some kind of pseudo-C#, it’s signature could look like

Functor<B> fmap<A, B>(Func<A, B> mapper, Functor<A> val) { ... }

For those who struggle to read that, this means that the function takes 2 arguments:

A function from type a to type b
A functor containing type a And returns a functor containing type b. The f in this type signature is the functor.

A simple example of a functor is a list. A list can:

Contain values of any type
Be mapped over with some function to produce a new list containing values of a different type

Another example are optional types which are prevalent in many functional languages. Optional types (Maybe in Haskell), either contain a value of some generic type, or containing no value at all. They are useful for writing functions that may or may not fail to produce a meaningful output. They are also functors since they can:

Contain one value of any type
Be mapped over with some function to produce an optional type containing one value of a different type. Concretely, if the optional type represents an actual value, we apply the function that value and wrap the result in another optional type. If the optional type contains no value, we do nothing to it.

Mapping over a data structure like this is a very common operation, so it makes sense to have an abstraction for it.

What is a monad

Note: This explanation may raise some eyebrows among those who know the topic well, but I really hate how most people are introduced to monads.

In one sentence, monads as they relate to programming are a design pattern that makes it simple to chain operations together while hiding a lot of the tedious boilerplate code involved in doing so.

More specifically, a monad is, like a functor, a data structure that contains values of some generic type. In fact, all monads are also functors. In addition to being functors, monads implement 2 useful operations which we’ll call pure and bind.

pure takes some value and returns that value wrapped inside of the monad. It is used to produce monadic values from non-monadic ones. pure has type

Monad m => a -> m a

In pseudo-C#, it would look like

Monad<A> pure<A>(A val) { ... }

bind takes as input a monadic value, and a function from a non-monadic value to a monadic one containing a different type, and returns a monadic value of the second type. In Haskell, it’s type is

Monad m => m a -> (a -> m b) -> m b

And as usual, in pseudo-C# it would look like:

Monad<B> bind<A, B>(Monad<A> val, Func<A, Monad<B>> binder) { ... }

Monads by example

The above explains briefly what a monad is, but not what it is useful for. Monads are used in functional languages to perform effectful computations without wanting to kill oneself. Examples of effectful computations are:

Computations that may or may not return a value (yep, optional types form a monad)
Computations that may read or write to a console or to disk
Computations which may write activity to a log
Computations which may return any amount of values
Computations which may interact with a piece of mutable state … and many more. Roughly, any time a computation is more than a simple deterministic mapping from an input to an output, we call it effectful.

Problems arise when trying to compose effectful operations in a purely functional way. To illustrate this, I’ll use the aforementiond example of writing activity to a log. I’ll use JavaScript for code demos since it is should be fairly easy to understand for people without too much knowledge in functional programming. Imagine we have a tiny program that does a sequence of numerical operations on a number, such as adding 5 to it, or multiplying it by 2:

function add5(a) {
  return a + 5
}

function mul2(a) {
  return a * 2
}

let res0 = 1
let res1 = add5(res0)
let res2 = mul2(res1)
let res3 = add5(res2)
console.log(res3)

This snippet declares a function for each of those 2 operations, and then performs them a few time. It calculates “((1 + 5) * 2) + 5” and prints the result - “17”. Now, let’s say we want to add logging to our program. In many imperative languages, one choice would be to have a global string variable for our log somewhere, and then have each function write to it, but in functional programming we avoid using mutable state. Therefore, the most straight forward way is to have each function return both the calculated number, and an string representing a log entry. Let’s implement that:

function add5(a) {
  return {val: a + 5, log: "Added 5\n"}
}

function mul2(a) {
  return {val: a * 2, log: "Multiplied by 2\n"}
}

let res0 = 1
let res1 = add5(res0)
let res2 = mul2(res1)
let res3 = add5(res2)
console.log(res3)

This program no longer produces the desired output! In a strongly typed language, it wouldn’t compile at all! The issue is that we are chaining functions, feeding the result of the first to the next, but each function returns a tuple but expects a single value as input. Essentially, add5 and mul2 have become effectful computations. Let’s fix that by implementing our aformentioned bind and pure operations:

function pure(val) {
  return {val: val, log:""}
}

function bind(input, binder) {
  let log1 = input.log
  let val1 = input.val
  let res = binder(val1)
  let log2 = res.log
  let val2 = res.val
  return {val: val2, log: log1 + log2}
}

function add5(a) {
  return {val: a + 5, log: "Added 5\n"}
}

function mul2(a) {
  return {val: a * 2, log: "Multiplied by 2\n"}
}

let res0 = pure(1)
let res1 = bind(res0, add5)
let res2 = bind(res1, mul2)
let res3 = bind(res2, add5)
console.log(res3)

Now, any time we want to use the result of one computation as the input to another, we can use bind to chain them. The program produces the desired result when run, a value and the final log:

{val: 17, log: 'Added 5\nMultiplied by 2\nAdded 5\n'}

It should be fairly clear to see how this is a pattern applicable in many cases beyond the one described here. For example, if the problem we were solving was instead that each of our math operations may fail to produce a value, we could swap out the bind and pure function but leave the using code exactly the same, and everything would still work as intended! In JavaScript, the code for that usecase might look something like, using null to represent the absense of a value:

function pure(val) {
  return val
}

function bind(input, binder) {
  if (input == null) {
    return null
  } else {
    return binder(input)
  }
}

Worth noting also is that the type contained within the monad (integers in the previous example) doesn’t have to always be the same for this to work. Let’s illustrate that with one last code example. We’ll add to our logging-example a function that converts a number to a string, and a function that concatenates a string with itself:

function toStr(a) {
  return {val: a.toString(), log: "Converted to string\n"}
}

function concat(a) {
  return {val: a + ", " + a, log: "Concatenated strings\n"}
}

And change the using code to:

let res0 = pure(1)
let res1 = bind(res0, add5)
let res2 = bind(res1, mul2)
let res3 = bind(res2, toStr)
let res4 = bind(res3, concat)
console.log(res4)

Which will now print

{
  log: "Added 5\nMultiplied by 2\nConverted to string\nConcatenated strings\n"
  val: "12, 12"
}

Notice how the exact same way of writing user code still functions perfectly even when mixing both integer and string types.

Notes

All monads are applicatives
All applicatives are functors
A monoid without a neutral element is called a semigroup

Worth noting that these are just a small sampling of useful abstractions plucked from category theory, and many more exist, though most of the literature on them is likely needlessly complicated for what they actually do. These are the ones I get asked about the most.

What does it mean that a monad is a monoid in the category of endofunctors

It’s not important. Stop asking.

If you really care, read this.

Links

https://wiki.haskell.org/All_About_Monads

http://learnyouahaskell.com/a-fistful-of-monads

https://stackoverflow.com/a/194207

Notes on Hindley-Milner type inference

2022-03-03T00:00:00+00:00

Motivation

I hate how hard type theory can be to get into. I’ve recently been fiddling with Hindley-Milner type systems and wanted to write down some notes that are hopefully understandable to future-me. If they are of use to anyone - cool, if not - also cool. I’ll definitely be referring to them in the future.

What and why

A Hindley-Milner (HM) type system is essentially an extension of typed lambda calculus with parametric polymorphism. This polymorphism is usually implicit and can importantly only be used at the ‘top-level’. It has some extremely nice properties such as the type of any term being inferrable from its use, and a fairly simple, almost built-in, method for inferring said types.

Types of types

HM deals with 2 kinds of type: Monotypes (aka. simple types) and polytypes (aka. type schemes).

Monotypes are simple, non-polymorphic types such as $Int$, $Bool$, $List(String)$ and so on. For simplicity, I’ll write concrete monotypes with uppercase first letter, and type variables with lowercase.

Polytypes, or type schemes, are universally quantified monotypes, such as $\forall a. List(a)$, which is concretely the type of a generic list, or $\forall a. a \xrightarrow{} a$, which represents the type of the identity function. Polytypes are, as the name implies, polymorphic - the $a$ in those types can be literally any type.

Instantiation and generalization

We can convert between monotypes and polytypes using instantiation and generalization.

Instantiation takes a polytype and returns a monotype by removing the quantifier and replacing all usages of it with some monotype. For example, $\forall a. a \xrightarrow{} a$ could be instantiated to either $Int \xrightarrow{} Int$ or $Bool \xrightarrow{} Bool$.

Generalization takes a monotype and returns a polytype by quantifying over all free type variables in the monotype. For example, $List(a)$ would generalize to $\forall a. List(a)$.

Importantly, the types $a$ and $\forall a. a$ are different. The former represents some concrete type (think $Int$ or $Bool$) which we don’t yet know, so must refer to with a variable. The latter represents literally any type, since it is polymorphic.

HM only allows universal quantification of types at the “topmost” layer. IE, any polytype must have the form $\forall [variables]. [type]$, with no nesting of the quantifiers. This, as far as I understand, makes the type system decidable, as opposed to the type system dubbed ‘System F’ which does not have this restriction.

Reading inference rules

The meat of HM can be specified concisely with a small set of inference rules. These look scary, but are fairly straight forward once you understand how to read them. Every inference rule has a set of premises and a conclusion, which are put on the top and bottom of a horizontal bar, like so:

\[\begin{prooftree} \AxiomC{P} \AxiomC{Q} \BinaryInfC{P $\land$ Q} \end{prooftree}\]

Which reads as “if I know P, and I know Q, then I know P and Q”. The rule has 2 premises and a single conclusion.

When dealing with actual type-theoretic inference, we often carry around an environment, usually written as $\Gamma$. The environment contains all the types of terms we have already inferred. Really, it is just a set of a terms and their corresponding types. The main operation we do on the environment is to make statements like $\Gamma \vdash x : \tau$ which means “given the information in the environment, we know $x$ is of type $\tau$”. This is called a typing judgment.

The ‘$\vdash$’ operator, called the turnstile, can be thought of a statement stating that a proof exists. Concretely, $\Gamma \vdash a$ means that given $\Gamma$, the environment, we can prove $a$, whatever that might be.

Inference rules for basic HM

Putting the previous knowledge to use, let’s look at the inference rules for HM, one by one.

Variable usage

The first rule describes how to assign types to usages of variables. It is written as:

\[\begin{prooftree} \AxiomC{$x : t \in \Gamma$} \UnaryInfC{$\Gamma \vdash x : t$} \end{prooftree}\]

In english: “If the environment actually contains the information that $x$ is of type $t$, then the environment proves that $x$ is of type $t$.”, which frankly is stating the obvious.

Interpretation: The environment keeps track of all the variables we have inferred the type of so far.

Function application

The next rules tells us how to type usages of functions in applications (function calls):

\[\begin{prooftree} \AxiomC{$\Gamma \vdash f : t_1 \xrightarrow{} t_2$} \AxiomC{$\Gamma \vdash a : t_1 $} \BinaryInfC{$\Gamma \vdash f(a) : t_2 $} \end{prooftree}\]

In english: “If the environment proves that $f$ is a function from type $t_1$ to $t_2$, and that $a$ is a expression of type $t_1$, then the environment proves that the application $f(a)$ is of type $t_2$”. This should be very reminiscent of modus-ponens for those familiar with propositional logic.

Interpretation: Giving the correct type of input to a function should result in the correct type of output.

Function abstraction

This rule tells us how to type function abstractions (declarations of functions, concretely as lambdas):

\[\begin{prooftree} \AxiomC{$\Gamma, x : t_1 \vdash e : t_2$} \UnaryInfC{$\Gamma \vdash (fun(x) \xrightarrow{} e) : t_1 \xrightarrow{} t_2 $} \end{prooftree}\]

In english: “If the environment and the assumption that $x$ is an expression of type $t_1$ prove that $e$ is an expression of type $t_2$, then the environment proves that the lambda $fun(x) \xrightarrow{} e$ has the type of a function from $t_1$ to $t_2$, written as $t_1 \xrightarrow{} t_2$.

Interpretation: A lambda expression is a function taking an input of some initially unknown type, returning a value of the type we infer body to, and the body may make use of the input.

This rule implies something about how we should infer the type of lambda expression. First, we make up a new, fresh type variable for the input parameter. Then we add info to the environment stating that the input parameter has that new type, and then we infer the type of the lambda body in that altered environment.

Let bindings

This rule tells us how to type let bindings of the form let x = e1 in e2 where x is an identifier and e1 and e2 are both expressions.

\[\begin{prooftree} \AxiomC{$\Gamma \vdash e_1 : t_1$} \AxiomC{$\Gamma, x : t_1 \vdash e_2 : t_2$} \BinaryInfC{$\Gamma \vdash (let \ x = e_1 \ in \ e_2) : t_2$} \end{prooftree}\]

In english: “If the environment proves that $e_1$ is an expression of type $t_1$, and the environment with the assumption that $x$ is an expression of type $t_1$ proves that $e_2$ is an expression of type $t_2$, then the environment proves that the let binding $let \ x = e_1 \ in \ e_2$ is of type $t_2$.”

Interpretation: Let bindings bind a name to a value in a following expression, and evaluate to a value of the type we infer that following expression to be. The following expression can naturally make use of the bound name.

As for lambda expressions, this implies that we need to alter the environment before inferring the type for the following expression.

If-then-else expression

This rule usually isn’t included in the minimal presentation of HM, but I include it since it is useful:

\[\begin{prooftree} \AxiomC{$\Gamma \vdash e_1 : Bool$} \AxiomC{$\Gamma \vdash e_2 : t_1$} \AxiomC{$\Gamma \vdash e_3 : t_1$} \TrinaryInfC{$\Gamma \vdash (if \ e_1 \ then \ e_2 \ else \ e_3) : t_1$} \end{prooftree}\]

In english: “If the environment proves that the condition expression of an if-then-else is of type $Bool$, and the environment proves that the expressions in both branches are of type $t_1$, then the environment proves that the expression $(if \ e_1 \ then \ e_2 \ else \ e_3)$ is of type $t_1$.”

Interpretation: The condition expression in an if-then-else must be a boolean, and the expression in both conditional branches must match it in type. The entire construct evaluate to said type.

Generalization and instantiation, again

What I have described so far is not the full set of inference rules. We are missing the rule for instantiation and generalization. I find these easier to explain and understand informally, so I have omitted them. Refer to the previous section on the topic for more info.

Unification and substitutions

When actually performing inference, unification is a process that often comes up. Unification is a process that takes as input 2 types, and returns a substitution that when applied to either type, will make the 2 types equal. We use unification whenever we want to constrain 2 inferred types to be the same.

A substitution is a list of bindings from names (of type variables) to concrete types. It can be applied to any structure containing types, which will replace the type variables in said type with the type in their entry of the substitution, where applicable.

As an example, the types $a \xrightarrow{} bool \xrightarrow{} list(a)$ and $int \xrightarrow{} b \xrightarrow{} c$ can be unified using the substitution $\{a=int, b=bool, c=list(int)\}$, yielding the unified expression $int \xrightarrow{} bool \xrightarrow{} list(int)$.

The typical algorithm for unifying 2 types is fairly simple, written below as F# code:

let rec unify (t1: Type) (t2: Type) : Map<string, Type> =
    match t1, t2 with
    | a, b when a = b -> Map.empty
    | TypeVar a, b when not (occurs a b) -> Map.ofList [(a, b)]
    | a, TypeVar b when not (occurs b a) -> Map.ofList [(b, a)]
    | TypeArrow (l1, r1), TypeArrow (l2, r2) ->
        let s1 = unify l1 l2
        let s2 = unify (apply s1 r1) (apply s2 r2)
        compose s1 s2
    | _ -> failwith "Could not unify types"

In english, this roughly means:

If the 2 types are the same yields the empty substitution
If either type is a type variable, and does not occur in the other type, return a substitution binding the type variable to the other type.
If we are unifying 2 function (arrow) types, unify the first type in each function type, apply the resulting substitution to the second type in each function type, and then unify those. Combine the 2 resulting substitutions into 1 and return that.
All other cases fail to unify. This indicates a type error in the program.

The so-called ‘occurs check’ is there to prevent pairs of types such as $a$ and $list(a)$ from unifying, since that would result in an infinite type.

The apply function takes a substitution and a type, and applies that substitution to a type.

Let-polymorphism

In HM, generalization happens only in 1 place: When inferring let bindings. This is known as let-polymorphism or let-generalization. This lets the type system handle expression such as:

let id = fun(x) -> x in (x true, x 3)

Where the name id has been bound to a polymorphic value of type $\forall a. a \xrightarrow{} a$. If one attempts to generalize in other places during inference, it will certainly cause issues and incorrectly inferred types.

This means concretely that, while we are inferring the type of a let expression, when we are about to put a new binding into the environment before inferring the body of the let, we generalize the type of that binding, and put the generalized type into the environment instead. Remember that generalization is just universally quantifying free type variables.

Typeclasses

Disclaimer: I don’t really know what I am talking about

HM is hard to extend without introducing a bunch of soundness holes. It can be done, though.

HM (as with many type systems) is essentially just glorified propositional logic. $p \xrightarrow{} q$ reads both as “the type of a function from p to q” in type-land and as “p implies q” in logic-land. Both are valid interpretations.

Haskell-style typeclasses under this interpretation are predicates. The Haskell type Num a => a -> a -> a written as predicate logic is $\forall a. Num(a) \xrightarrow{} a \xrightarrow{} a \xrightarrow{} a$.

One can extend the type scheme, which was previously the most general way to describe a type at our disposal, to also include a list of predicates. Each predicate could be represented as a tuple of an identifier and a type, indicating that the type ‘is a member of’ the typeclass described by the identifier. If one accumulates these predicates during inference, just as is done for substitutions, one can solve these constraints in the end to see if the program typechecks. The problem basically becomes a typical predicate-logic problem, and one that languages such as Prolog interestingly seem like they were made to solve.

Much more about these ideas in the “Typing Haskell in Haskell” paper linked to below.

Links

https://github.com/pema99/bonk/blob/master/src/Inference.fs

https://course.ccs.neu.edu/cs4410sp19/lec_type-inference_notes.html

http://dev.stephendiehl.com/fun/006_hindley_milner.html#inference-monad

https://jgbm.github.io/eecs662f17/Notes-on-HM.html

https://gist.github.com/chrisdone/0075a16b32bfd4f62b7b

pema.dev

Mipmap selection in too much detail

Mipmapping primer

Pixel derivatives

Derivatives to mipmap levels

What does the mapping look like conceptually?

What does the hardware actually do?

Exploring the full 4D mapping function

Reverse engineering the hardware implementation

Missing elliptical transformation

Side quest: Texture filtering theory and vector calculus

Understanding the elliptical transformation

Trilinear filtering

Anisotropic filtering

Bonus: More visualizations

Bonus: Nvidia’s approximation

Closing words

Haskell is not category theory

What, why, who?

What is a category?

\(Hask\) is a category

\(Hask\) is not a category

Functor is not \(Functor\)

A note on kinds of functors

Natural transformations

Monad is almost \(Monad\)

Monoid in the category of endofunctors

Don’t break the law

Some links

6 interesting languages and their selling points

Foreword

Futhark

Selling point: Effortless, functional GPGPU

ISPC

Selling point: A programming model focused on vectorization

Koka

Selling point: Algebraic effects

Unison

Selling point: A language without source code

APL

Selling point: Terseness taken to the extreme

Prolog

Selling point: Logic programming

Blogs I love

Functional programming

Unity stuff

General tech

Techno wizardry

Shaderception - a deep dive

Intro

The language

The compiler

On UdonSharp

Immediate mode compilation

Aggresive inlining

Extern calls

Label linking and register allocation

Producing a binary

The virtual machine

Stack or register based?

The Shaderception instruction set

1 - PUSHCONST (float4)

2 - PUSHVAR (id)

3 - BINOP (op)

4 - UNOP (op)

5 - CALL (id)

6 - SETVAR (id)

7 - JUMP (loc)

8 - CONDJUMP (loc)

Executing the instructions

Dynamic typing is cursed

NaN propagation is cursed

How not to crash your GPU driver

The tooling

The Shaderception world

User submitted programs

Shaderception cross

Shaderception standalone

Shaderception VSCode syntax

Weird HLSL tricks Unity doesn't want you to know about

`Functor` is not \(Functor\)

`Monad` is almost \(Monad\)

`1 - PUSHCONST (float4)`

`2 - PUSHVAR (id)`

`3 - BINOP (op)`

`4 - UNOP (op)`

`5 - CALL (id)`

`6 - SETVAR (id)`

`7 - JUMP (loc)`

`8 - CONDJUMP (loc)`

`isnan()` is a lie

`[forcecase]` is great