listing
All programs work with information. To the programmer, data takes many forms: a list, a table, an image, an essay, an album. To the computer, it all looks basically alike: it’s just a bunch of bytes in memory. Imagine a single column of a spreadsheet, each of the cells filled with a number from 0 to 255. All your stuff is scattered across that column, along with bookkeeping to track what lives where.
A friend of mine was recently given an interview task: a spec for a simple command-line tool, along with ten or so increasingly fiddly feature requests, in case of any free time before the two-hour deadline. I wanted to know if ChatGPT’s O3 model was hireable, so I handed over the PDF and got back an answer. As someone who’s barely written a line of Rust in my life, the thing was done in about twenty minutes; most of that was me checking it all worked as promised.
I’ve been working on a new programming language, called Raven. It’s still experimental, and not ready for serious (or even silly) users. But curious language nerds such as yourself may find it interesting. If you want to follow along then sponsor me, and I’ll give you early access to the repo along with regular behind-the-scenes updates.
Raven is small but smart. It should feel tight and intuitive, lightweight enough for interactive notebooks and scripting, yet capable and adaptable for big projects with hard constraints.
The Zig language begins with a barebones C-like system. Functions, structs, numbers and pointers are standard. Tagged unions, some syntax sugar (try, defer) and safer defaults alone make Zig a pleasant but unremarkable improvement, like many of the C challengers that seem to be cropping up lately. What does make Zig remarkable is its compile-time evaluation. You can run basically any code at build time, and that gives you a lot of C++’s templating power without feeling nearly as complex.
WebAssembly is an odd compiler target, because of its unusual mix of high- and low-level style. It gives you pointers and a flat array for your heap memory, yet no access to the program stack. It uses jump instructions for control flow, while only allowing the structured kind. It has local variables and even expressions, but instructions take and produce values through a data stack.
There are well-known ways to deal with most of these peculiarities: you can create a shadow stack to model C-like stack pointers, or turn jump-based control flow back into loops and conditions with the “stackifier” algorithm.
Last month a startup called Modular released a new language called Mojo (not to be confused with the existing indigenous one). Based on Python and designed for ML hardware and models, Mojo’s goals (“the usability of Python with the performance of C”) coincide with those of the Julia language, so I’m interested to compare notes.1
Some throat-clearing to start. I’ve done a bit of work on Julia, particularly on its support for ML hardware and models, though it’s a few years since I’ve been directly affiliated.
Most programming languages conflate the building of data structures and the modelling of information.
Of course, you need data structures to store information, but there are two different levels of abstraction. On one level are pointers and structs and B-Trees and so on, on another the user models of a dictionary, list or relational table, built on the first. The former uses finicky, low-level tools to create the neat internal logic of the system.
Conversations with ChatGPT, a recently released chatbot, reportedly cost its inventors at OpenAI a few cents each. By Internet standards this is shockingly expensive. Though it’s touted as the future of the search engine, any company scaling the technology up to the world’s 10 billion or so daily queries will face suffocating costs. The milestone for Artificial Intelligence may be that it’s now about as expensive as the real thing: Amazon’s mechanical turk service (which Jeff Bezos called “artificial artificial intelligence”) also pays its human labourers a few cents per question-response task.
Three years and two months after it launched on Kickstarter, copies of social deduction game Blood on the Clocktower are finally arriving in the UK. During the long wait eager players have been able to backstab each other using print-and-play copies and the online helper (particularly welcome during lockdowns). But it’s nice to have the real thing in our hands. I’m here to commemorate the occassion with statistics, and in particular to show the ranking system I’m using to see how well my players are doing – just for fun, of course.
I just put up a draft paper with a proposal for mixing imperative and functional styles of programming. The gist of it is that we can write code like this:
fn arange(start, end) { result = [] for i in start:end { push(&result, i) } return result } or this:
fn matmul(A, B) { [m, n] = size(A) [n, p] = size(B) C = zeros(m, p) for i in 1:m, j in 1:p { sum = 0 for k in 1:n { sum += A[i, k]*B[k, j] } C[i, j] = sum } return C } and these can in fact be functional programs (at least for any definition of “functional programming” that isn’t based on syntax).
The website thispersondoesnotexist.com shows off images created by StyleGAN, a kind of machine learning model trained on around 70,000 headshots from Flickr. Although the photos on the site resemble real ones, they are wholly made up: StyleGAN can generate an unlimited number of convincing faces on the fly, each unique and never to be seen again.
Under the hood, StyleGAN starts with a set of 512 numbers, known as the “latent state”, like:
It’s easy to mock NFTs (non-fungible tokens),1 or even to feel cynical about them. Cryptocurrency enthusiasts are spending enough cash to buy a rather nice house, and instead receiving a signed URL for a digital work of art, like this one.
That pièce de résistance is called CryptoPunk 5822. As the name suggests, it is part of an exclusive collection: just 10,000 similar ones exist on the blockchain. Each CryptoPunk is randomly generated, and only 481 of them sport that distinguished bandana, making this one unusually rare.
So you want to show off the size of your dictionary? Look no further than this quick test, which will work out how many words you know.
travel which word has the closest meaning? friendship journey answer researcher Just click the word that’s closest in meaning to travel (n.). As you play we’ll narrow down the estimate of your vocabulary size, and you can stop whenever you get bored. Right now we think you know about 2,000 words (between 400 and 200,000).
“Words can be like X-rays if you use them properly – they’ll go through anything.”
Wordle is a neat little word game, somewhere between mastermind and hangman. Try to guess the hidden five-letter word and it will tell you which letters are correct, and whether they are in the right position. You have six guesses to narrow down the right answer. Can we work out an automated strategy for this game?
Roland Barthes’ essay on “The Death of the Author” is inscrutable to this reader. The guy seems determined to use obscure French names in place of regular verbs and nouns as much as possible. That is a shame, because – as so often – the Scholarly Writing obscures an interesting idea.
The question is whether it matters what writers think about their own stories. Many would say yes, of course it does: the author knows what really happened after that cliffhanger, but just didn’t write it down.
Mistakes slip into publications of all kinds. If you’re a news organisation, you can mitigate this with a rigorous editing process. If you’re a blogger, it’s hard to balance perfecting your words with publishing them.
Digital media opens a new option. After checking for typos and obvious errors, just publish, and let your audience be your editor. This is especially useful for problems that are hard to catch ahead of time, like ambiguous wording.
An astronomer might say that blue is the warmest colour, and red the coolest. Lukewarm stars, at a temperature of around 3500K, give a gentle red glow. Truly hot ones, at about 6000K, become brilliant white, and above that blue and ultraviolet. Yet in most cultures red and orange stand in for heat, while white and blue mean cold. If this rule isn’t given by the laws of physics, how do we learn it?
The goal of statistical inference is to tell us things that we didn’t know based on things we do. The things want to learn about could be anything – phylogenetic trees, for example – but usually we’ll talk about numerical summaries, like “the GDP of France”, or “the reproductive rate of covid-19”. In mathematical notation we’ll label the set of numbers we don’t know $[\theta \in \mathbb R^n] (as in, $[\theta] is a list of $[n] unknown numbers) and the numbers we do know $[X \in \mathbb R^m].
Here’s a plot of Covid-19’s reproductive rate ($[R]) over time in the UK, based on daily case numbers and seroprevalence surveys. Because the estimates are uncertain, each light blue line on the plot shows a possible path that $[R] could have taken, with all paths together giving a sense of what’s plausible on any given day.
Estimates of the reproductive rate in the UK, up to 16th December 2020. Early in the year $[R] could be almost anything, given the limited data at that time.
Throughout history, people have used coded language – hiding secrets in plain sight – for self protection. Codes are often invented for purpose, but can also develop of their own accord. Take Polari, spoken by gay men in Britain up to the 1960s. In “Fabulosa!”, Paul Baker explains how the language developed from other forms of slang, and how it was used to obscure then-criminal practice from a frowning public.
Grouchy passengers on the London Underground are offered all kinds of stress-reducing therapies. From time immemorial, the tube’s fluorescent-lit carriages have been tiled with images of crystal clear oceans, comprehensive holiday packages, and sharp-suited individuals swearing by particular vitamin pills. But a more recent prescription is a very old one: meditation, a Buddhist practice dating back to the fourth century BC.
Meditation has not, historically, been a very lucrative industry. The practitioner’s biggest outlay might be for their Zafu, a round floor cushion.
Mjolnir is, one the one hand, a library implementation of Julia’s type inference engine. On the other, it’s a version of the computation-graph-tracing technique used by JAX.1 Those might seem like quite different things, but they’re really two sides of the same coin. Understanding that relationship is enlightening in itself, but more interestingly, it leads to some ideas for getting the best of both systems.
Abstract Interpretation Julia uses an abstract interpretation approach to type inference.
Clojure has introduced a very interesting idea called ‘transducers’, which decouple sequence transformations (mapping, filtering etc) from sequence implementations like vectors, lazy lists and channels. Transducers and their benefits are well-covered elsewhere, but I’d like to explore some tradeoffs, and compare an alternative (and extremely hypothetical) design based on a staple of the functional programming world, effect handlers.
There are many useful operations that we can carry out on sequences, like mapping, filtering, interleaving, partitioning and so on.
Programming language design and implementation is, always has been, and likely always will be, hard. But it’s hard today for completely different reasons to those fifty years ago, when the PL field first bloomed. In the past a budding language designer more or less just had to hack together a parser, throw out machine code and call it a day (albeit this is harder than it sounds with then-current tools). In 2020 such a language could be a weekend project – but in turn, language designers face many new challenges and a much higher bar for success.
When you write a for loop in Julia, like
for x in xs # do something with x end this expands to
next = iterate(xs) while next != nothing x, state = next # do something with x next = iterate(xs, state) end Any Julia object – an array, a linked list, an infinite generator, whatever – can then implement iterate(xs) and iterate(xs, state) to make iterating over it with for possible, as well as getting a few other utilities like collect.
Julia programmers love nothing like a good polysemy. One of the more extreme examples is the community’s use of the word “type”, which refers to a cluster of related, but subtly distinct, features and their roles in the language. Ambiguity can provide a useful shorthand, but it can also be confusing to newcomers and those outside the community, so it might be useful to disambiguate a bit.
In short, types are Julia values that can play in one of the following parts: as tags for identifying meaning, selectors for dynamic dispatch, descriptions of data layout, annotations provided by type inference, hints for compiler specialisation, and constraints for compile-time correctness checking.
Originally posted on the Julia website.
Any sufficiently complicated machine learning system contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of a programming language.1
As programming languages (PL) people, we have watched with great interest as machine learning (ML) has exploded – and with it, the complexity of ML models and the frameworks people are using to build them. State-of-the-art models are increasingly programs, with support for programming constructs like loops and recursion, and this brings out many interesting issues in the tools we use to create them – that is, programming languages.
Can the Universe Simulate Itself?
I’m not merely asking if we can simulate the laws of Physics, nor whether we can simulate them with such fidelity that they give rise to life. Those are pansy questions.
I’m asking if the universe can simulate itself exactly – so exactly that, if the simulation were left running long enough, it would give rise to humans just like us, who would then initiate the same simulation!
Have you ever seen a programming language like this?
++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.> ---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++. What you’re looking at is the “hello, world!” program of a language called brainfuck. Brainfuck is a sort of postmodern approach to language design, the kind of thing that Jackson Pollock’s evil digital twin would have used. It has only eight instructions for moving around an array of bytes, adding and subtracting one as you go. Yet surprisingly enough, it’s just as powerful as a full programming language like Python.
Julia has a library called CUDAnative, which hacks the compiler to run your code on GPUs.
using CuArrays, CUDAnative xs, ys, zs = CuArray(rand(1024)), CuArray(rand(1024)), CuArray(zeros(1024)) function kernel_vadd(out, a, b) i = (blockIdx().x-1) * blockDim().x + threadIdx().x out[i] = a[i] + b[i] return end @cuda (1, length(xs)) kernel_vadd(zs, xs, ys) @assert zs == xs + ys Is this better than writing CUDA C? At first, it’s easy to mistake this for simple syntactic convenience, but I’m convinced that it brings something fundamentally new to the table.
“Any sufficiently advanced technology is indistinguishable from magic,” go the famous words of Arthur C. Clarke. Often with seemingly-magical technology, a small dose of knowledge is enough to wash away our wide-eyed awe and bring us back to sweet, comfortable cynicism. Yet some ideas don’t lose their lustre so easily; it can be a little hard to believe they work even once you understand them.
Laziness is one such idea. Today I’d like to give you a taste of the borderline-magical things that lazy sequences can do.
This post gives an overview of working with larger-than-memory data in the Julia language. I don’t have any particular use for this, but was curious about the problem as the current tools for out-of-core analysis seem to be pretty lacking. I’ve attempted to come up with a basic implementation of an array backed by persistent storage, and this post explains how I’ve put it together.
Aside from any insights about the problem itself, I’m going to write a lot about some of Julia’s more unusual features and how they lock together.
Computers are more powerful than ever before. Now that you can buy watches which outpace the earliest supercomputers, it seems logical to think that the efficiency of your software no longer matters. And yet, performance is the new black, with recent languages like Go, Julia and Rust aiming to provide expressive power and ease of use without compromising on CPU cycles.
Julia is probably the most extreme example of this trend, with its lofty goal of being as dynamic and expressive as languages like Python, Ruby or R while still traveling at the speed of C.