Normalisation of Deviance, Oil Spills, AI, Syntax Highlighting

I was looking at https://danluu.com/wat/, titled "normalisation of deviance" in the article, which the author described as "what we can learn from aviation, civil engineering, healthcare, and other safety critical fields".

The phrase "normalisation of deviance" comes up a lot when talking about how things went so wrong, for example the deepwater horizon oil spill by BP in 2010. Often the phrase implies doing something more and more risky, and assuming previous (often lucky) success will continue. I think that this particular oil rig had a pretty decent safety record[citation needed] before this spill, for example.

There's a magazine called hindsight (with previous editions available here) primarily about air traffic management, a field that generally cares about safety and makes a pretty good job of it. Planes collide a lot less than the average computer program fails, for example.

One thing I read in a previous edition was about neural networks being used to plan routes for planes, and experiments about that and so on. A neural network is a process that tries to optimise something, which means it pushes the boundaries of the constraints imposed on it. In the example presented, the computer came up with flight paths that required complex manoeuvres by pilots, and the computer-generated solution was ignored or otherwise discarded. I can't remember the exact context of this, so I don't know if it was "live" or anything, but I found it interesting since it's a good example of that kind of thing, which I think happens a lot. If you optimise for something, then that means you're willing to sacrifice everything else (unless you start optimising for that too).

Anyway, in the latest edition [PDF] there's more stuff on AI, including the following from page 14:

Key Points:

I recommend reading the article, it's interesting, but I think these points are worth thinking about in a more general sense.

This reminds me of Cory Doctorow's "Reverse centaurs and the failure of AI", wherein a "centaur" is a human who is using a computer or machine to make them more effective, and a reverse centaur is a human doing menial work supervising an "automated" system.

As far as programming goes, the obvious Big Thing in AI-assistance is github's copilot, which I've never used and don't care about — apparently legal action is being taken. But there is something I care about, which is a bit more mundane: syntax highlighting.

Specifically, I was talking with a friend recently, about why having errors in a text editor felt annoying, but writing code that doesn't get highlighted because you typed it in a way which doesn't match e.g. the form of an integer literal felt like a healthy collaboration with the software.

I think the reason is basically that you can trust the computer when it says you're right (up to something objective and measurable like "matches the specification for int literals"), but you can't trust it when it says you're wrong. For example I don't use spell checkers while typing because I use words which aren't in the dictionary, and some words I spell in a nonstandard way deliberately. If the computer tells me I'm wrong, by underlining a word in red, it isn't always accurate.

I wrote a syntax highlighter for hare a while ago, and in this commit I implemented something with the goal that someone using the highlighter would be able to have good feedback on the correctness of their float literals without annoying false-negatives. Float literals in hare are split into two parts, the 1.123 part, and an optional suffix, f, f32, or f64.

(It's slightly more complex but it doesn't matter).

I think it would be annoying to have a valid float literal, start adding a suffix, and then have the computer tell you that your code is no longer correct when you're halfway through writing f64. The change I made in that commit means that the 1.123 part of the float stays highlighted while you're halfway through adding the suffix. This makes sense because the correctness of the 1.123 part and the correctness of the f32 part are seperate concerns.

So, looking to the future, what more is there to do? How can a text editor convey useful information? I'm writing a text editor now, but I'm not at the point where I can experiment with what I'd like to. I want to try things like "only highlight a variable if the name is in scope", or "only highlight a function if it's available from the included header files". The technical part of this already exists (at least I think e.g. clangd should be suitable) but I still need to write the rest of the editor.

Further reading: