I like the idea of Nix, but don't enjoy using it

I’ve been playing with Nix and NixOS a lot more lately. I installed NixOS on one of my servers, I installed the Nix CLI on my laptop, I tried to use Nix to [build a Docker (BROKEN) image], I use Nix flakes.

This post was written from the perspective of a person new to Nix, but experienced with other computer languages. Thus, it’s probable that I might be doing something wrong or maybe complaining about something that’s obvious to you. However, these are issues that others may face.

It was also written over several months as I gathered issues, so even looking back, I see mistakes.

The Good Parts

The biggest advantage about Nix is that I can define an entire system using a text based language that I can check into a Git repo. The workflow of identifying a bug, making a change, running nixos-rebuild switch, verifying it works, then checking it in. Then six months from now being able to Git blame and see why I made a change is great.

A screenshot from GitHub showing a simple Nix script change to enable json based logs in Docker. The diff clearly shows the change vs just arbitrarily changing config on a server.

The Bad Parts

Nix, the language

The Nix language is not-intuitive. I understand it’s not like most computing languages that are designed for performing operations and it’s intended to be a declarative configuration model, but as a developer with skills in other languages, I can’t

Variables

To declare a variable, you have to do this:

1
2
3
4

let
  x = 123
in
  blah

My problem is I might start with some code like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
  pkgs ? import <nixpkgs> {},
  content ? import ./website.nix {},
}:

{
  nginx = pkgs.nginxStable.overrideAttrs (oldAttrs: {
    # ...
  });

  nginxConfig = pkgs.writeText "nginx.conf" ''
    # blah
  '';
}

But if I want to refer to nginx in nginxConfig, I have to radically change this code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
  pkgs ? import <nixpkgs> {},
  content ? import ./website.nix {},
}:

let
  nginx = pkgs.nginxStable.overrideAttrs (oldAttrs: {
    # ...
  });
{
  nginx = nginx;
  nginxConfig = pkgs.writeText "nginx.conf" ''
    # blah
  '';
}

This requires a big change in the code to do something that is trivial in every other programming language.

Scoping

Nix, the CLI

Errors

Having useful error messages is critical for developers, but nix does not give good error messages. Now, a lot of this is because I’m new to Nix and make lots of dumb mistakes when coding, but a language needs to be approachable by new people. Otherwise, they can’t turn into experienced devs.

Let’s see an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

error:
   … while calling the 'derivationStrict' builtin
	 at <nix/derivation-internal.nix>:34:12:
	   33|
	   34|   strict = derivationStrict drvAttrs;
		 |            ^
	   35|

   … while evaluating derivation 'hugo-nginx-image.tar.gz'
	 whose name attribute is located at /nix/store/vfvjcnwvwy1lrjfaz0wvvd5fwcf5gb4v-nixpkgs/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:336:7

   … while evaluating attribute 'buildCommand' of derivation 'hugo-nginx-image.tar.gz'
	 at /nix/store/vfvjcnwvwy1lrjfaz0wvvd5fwcf5gb4v-nixpkgs/nixpkgs/pkgs/build-support/trivial-builders/default.nix:59:17:
	   58|         enableParallelBuilding = true;
	   59|         inherit buildCommand name;
		 |                 ^
	   60|         passAsFile = [ "buildCommand" ]

   (stack trace truncated; use '--show-trace' to show the full, detailed trace)

   error: expected a set but found a function: «lambda @ ~/projects/technowizardry.net/nix/docker.nix:1:1»

A file that I wrote was only mentioned at the very last point. Let’s zoom in:

1
2
3
4
5
6
7
8
9

{
  pkgs ? import <nixpkgs> {},
  docker ? import ./nix/docker.nix,
  hugo ? import ./nix/hugo.nix {},
}:

pkgs.dockerTools.buildLayeredImage {
  # ...
}

Can you spot the mistake? I spent time hunting through the buildLayeredImage block, but that was entirely irrelevant. The error message seems to suggest line 1 and col 1, but it’s only a {.

Turns out, the fix is (note the curly braces).

1
2

-  docker ? import ./nix/docker.nix,
+  docker ? import ./nix/docker.nix {},

It was not intuitive for me to figure that out.

Confusing Commands

There’s quite a few Nix commands:

  • nix
  • nix-build
  • nix-channel
  • nix-collect-garbage
  • nix-copy-closure
  • nix-daemon
  • nix-env
  • nix-hash
  • nix-info
  • nix-instantiate
  • nix-prefetch-url
  • nix-shell
  • nix-store
  • nixos-build-vms
  • nixos-container
  • nixos-enter
  • nixos-firewall-tool
  • nixos-generate-config
  • nixos-help
  • nixos-install
  • nixos-option
  • nixos-rebuild
  • nixos-version

Now you can probably guess what most of these do, but that’s not what I mean. I want to upgrade my Nix daemon. How do I do it? nix upgrade? Nope.

Nix, the package manager

What I would have expected for versioning

As I understand it, when you pick a package from nixpkgs (e.g. pkgs.nginx), you’re picking what ever happens to be defined as the version in the HEAD commit in NixOS/nixpkgs repo. Using a Nix Flake can freeze the commit you’re looking at, but that is all of nixpkgs. You’re still at the mercy of whatever the nixpkg maintainer used as a version. That version could be out of date or it could even be an unofficial release candidate (e.g. this commit adopted an rc version). Or maybe the docker uses 27.x which is not officially supported by your Kubernetes engine RKE1.

What are my options? With some packages like Docker, Nixpkgs continues to maintain the old versions, so I can just change from pkgs.docker to pkgs.docker_26, but not every package has this luxury. Nor is it clear how I actually use it when I use:

1

virtualisation.docker.enable = true;

In the onedrive example, the change was unexpected. It appears that it’s only a release candidate in the unstable Nix channel, but unstable seems to be the default.

I could always just copy the derivation into my own repository and never deal with any issues, but that’s non-trivial.

I see the nixpkgs style of versioning to be fundamentally not user-friendly. I think I’d prefer something like programming language dependency modeling (think Ruby Bundler’s Gemfile or Python’s Pipfile) where I can do:

1
2

containerd =~ 24.0.0
firefox = *

Right now, the upgrade story brings risk for unexpected changes. I do a nix flake up and nixos-rebuild switch and things just poof change.

How do I even override something?

I’m trying to use the Kubernetes NixPkg, but I needed to modify it to:

  • Pin the control plane and worker versions so I can bump the control plane before bumping the worker nodes
  • Pin the pause image so it doesn’t get accidentally deleted breaking my cluster
  • Not delete the CNI binaries automatically every time Kubelet starts

How do I pin NixPkgs?

To pin the versions, I found posts that said I could just import a specific commit of NixPkgs like this and then update the commit as I desire:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

diff --git a/flake.nix b/flake.nix  
index a4f5dc0..2ff8429 100644  
--- a/flake.nix  
+++ b/flake.nix  
@@ -3,6 +3,8 @@  
   
  inputs = {  
    nixpkgs.url = "github:nixos/nixpkgs";  
+    # Grab a specific version of the NixPKGs so we don't accidentally update K8s  
+    nixpkgs-k8s.url = "github:NixOS/nixpkgs/78324291425a318af7b6fe08ce0646291f5588db";  
    nixos-hardware.url = "github:NixOS/nixos-hardware/master";  
    disko = {  
      url = "github:nix-community/disko";  
@@ -10,7 +12,7 @@  
    };  
  };  
   
-  outputs = { self, nixpkgs, disko, ... }@inputs: {  
+  outputs = { self, nixpkgs, nixpkgs-k8s, disko, ... }@inputs: {  
    nixosConfigurations.srv5 = nixpkgs.lib.nixosSystem {  
      specialArgs = {inherit inputs;};  
      modules = [  
diff --git a/parts/kubernetes.nix b/parts/kubernetes.nix  
index 5947840..47207ed 100644  
--- a/parts/kubernetes.nix  
+++ b/parts/kubernetes.nix  
@@ -1,4 +1,4 @@  
-{ config, lib, pkgs, ... }:  
+{ config, lib, pkgs, nixpkgs-k8s, ... }:

It’s not exactly what I want which is to pin to a specific Kubernetes Major.Minor versions, but it’s fine. However, it still doesn’t make any sense:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

{ config, lib, pkgs, nixpkgs-k8s, ... }:
let
  # ...
in
{
  services.kubernetes = {
    kubelet = {
      # ... How does one specify that kubelet is from nixpkgs-k8s, not the default pkgs?
    };
  };
}

What I think is happening is that the nixpkgs-k8s parameter does point to the expected commit, but nixpkgs is composed of both configuration and packages. The configuration services.kubernetes.kubelet works off the latest commit.

My first assumption, until I was actually writing this post, was that I’d have to duplicate the services/cluster/kubernetes files and here where it builds the systemd service, I somehow modify the path to kubelet from ${top.package}/bin/kubelet to ${nixpkgs-k8s.services.kubernetes.package}

However, I now see a services.kubernetes.package option that I think I can override. But how? Maybe nixpkgs-k8s.services.kubernetes.package?

1
2
3
4
5
6
7

{ config, lib, pkgs, nixpkgs-k8s, ... }:
let
  # ...
in
{
  services.kubernetes = {
    package = nixpkgs-k8s.services.kubernetes.package;

Nope, I get:

1

error: attribute 'nixpkgs-k8s' missing

I’m not really sure what to do to fix this. Maybe I need to vendor the entire kubernetes/ folder myself, but I look at it and I will have to make a number of modifications to get it to work to change variable references.

Conclusion

Nix is a fascinating idea. The ability to declaratively define my entire OS is great for servers. I can have all my servers look exactly the same and easily create ones as I want. I like the idea of being able to Git revision control changes. It really satisfies my itch to centralize and cleanly define configuration.

However, I find working with the Nix language and Nix the package manager to be extremely frustrating. Usually there’s a learning curve that I can get over and become proficient enough at it, but Nix I struggle with. I’m not sure what my threshold for, it’s too difficult to be worth it is.

Copyright - All Rights Reserved

Comments

To give feedback, send an email to adam [at] this website url.

Donate

If you've found these posts helpful and would like to support this work directly, your contribution would be appreciated and enable me to dedicate more time to creating future posts. Thank you for joining me!

Donate to my blog