Techno Wizardry

Trying to use LiteLLM Proxy in my smart home

Mon, 02 Mar 2026 10:00:00 -0800

Everybody’s doing it. I guess I need to do an AI, too. In my home, I have a few different tools that use generative AI and LLMs. I talk to my Home Assistant Voice Preview voice assistants which leverage a self-hosted Ollama running llama3.2. I use OpenWebUI, tried Tabby as an experimental coding assistant. I use DeepInfra for larger models that don’t fit on my own GPU.

However, my problem is that each program supports different providers and models. Some support OpenAI style APIs to any provider, some only support Ollama APIs. If I wanted to forward my Home Assistant queries to DeepInfra, it wasn’t easy to do because there wasn’t an integration. If I wanted to change the model that Tabby uses between different models, I had to redeploy the service.

What I wanted is a way was to be able to support both Ollama and OpenAI clients and be able to forward requests to different upstream providers based on policy.

My Requirements

I had a few different clients that connected to LLM providers:

Home Assistant - Supports OpenAI, Ollama, etc.
Open WebUI - Supports OpenAI compatible and Ollama
Tabby - Coding assistant
My various test projects

I had single NVIDIA GeForce 1080 Ti and NVIDIA GeForce 2080 at home which was fine for some work, but would take too long to respond to my voice assistant. Home Assistant did support OpenAI, but didn’t support a custom OpenAI endpoint so I couldn’t redirect it to a paid-for LLM provider, such as DeepInfra or OpenRouter. I wanted the flexibility to send any client to any provider without worrying about API compatibility or API keys.

I was looking for some kind of self-hosted LLM call router that could route requests.

The landscape

Some initial research showed that there were a number of projects

ArchGW
LangFuse
Helicone
LiteLLM

The Langfuse Helm chart wanted to deploy 3x Apache Zookeeper, 3x Clickhouse, Redis, MinIO, a web app, and a worker. While I could cut the number of replicas, that was too much for a home lab that had TPS in the order of <5 request per hour.

ArchGW provided a way to route calls based on a fast AI analysis of the prompt, but I couldn’t get it to route in my testing. A model alias seemed simpler.

Helicone was focused on observability–how long do prompts take to query, etc. Cool, but not what I need

LiteLLM

Why LiteLLM? LiteLLM seemed simple enough and do what I wanted, but little did I know it was going to be a giant pain.

Tool calls are breaking my Home Assistant

Home Assistant needed to call via the Ollama API to LiteLLM, but LiteLLM didn’t natively support Ollama. I looked for an Ollama-OpenAI proxy and found [this one].

After adding it to Home Assistant and trying the chat feature, I’m faced with an opaque error.

A Wireshark packet capture shows several proxy calls:

The final call to Ollama shows this request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


{
 "n": 1,
 "model": "llama3.2:latest",
 "top_p": 1,
 "stream": false,
 "messages": [
 {
 "role": "user",
 "content": "### System:\nYou are a voice assistant for Home Assistant.\nAnswer questions about the world truthfully.\nAnswer in plain text. Keep it simple and to the point.\nWhen controlling Home Assistant always call the intent tools. Use HassTurnOn to lock and HassTurnOff to unlock a lock. When controlling a device, prefer passing just name and domain. When controlling an area, prefer passing just area name and domain.\nWhen a user asks to turn on all devices of a specific type, ask user to specify an area, unless there is only one device of that type.\nThis device is not able to start timers. Produce JSON OUTPUT ONLY! Adhere to this format {\"name\": \"function_name\", \"arguments\":{\"argument_name\": \"argument_value\"}} The following functions are available to you:\n{'type': 'function', 'function': {'name': 'HassTurnOn', 'description': 'Turns on/opens a device or entity', 'parameters': {'type': 'object', 'required': [], 'properties': {'name': {'type': 'string'}, 'area': {'type': 'string'}, 'floor': {'type': ... (litellm_truncated 11355 chars)"
 }
 ],
 "temperature": 1,
 "presence_penalty": 0,
 "frequency_penalty": 0
}

Curious. What is this:

1

Produce JSON OUTPUT ONLY! Adhere to this format {\"name\": \"function_name\", \"arguments\":{\"argument_name\": \"argument_value\"}} The following functions are available to you:\n{'type': 'function', 'function': {'name': 'HassTurnOn', 'description': 'Turns on/opens a device or entity'

Somewhere, the request with a native structured tool call is getting turned into a textual message that we hope the model can understand. The response gets passed back as a JSON serialized as a string, not an actual tool call.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


chatcmpl-0a38a5eb-da8b-4485-92e7-91eef8955916

{
 "n": 1,
 "model": "llama3.2:latest",
 "top_p": 1,
 "stream": true,
 "messages": [
 {
 "role": "user",
 "content": "### System:\nYou are a voice assistant for Home Assistant.\nAnswer questions about the world truthfully.\nAnswer in plain text. Keep it simple and to the point.\nWhen controlling Home Assistant always call the intent tools. Use HassTurnOn to lock and HassTurnOff to unlock a lock. When controlling a device, prefer passing just name and domain. When controlling an area, prefer passing just area name and domain.\nWhen a user asks to turn on all devices of a specific type, ask user to specify an area, unless there is only one device of that type.\nThis device is not able to start timers.\nYou ARE equipped to answer questions about the current state of\nthe home using the `GetLiveContext` tool. This is a primary function. Do not state you lack the\nfunctionality if the question requires live data.\nIf the user asks about device existence/type (e.g., \"Do I have lights in the bedroom?\"): Answer\nfrom the static context below.\nIf the user asks about the CURRENT state, value, or mode (e.g., \"Is the lock l... (litellm_truncated 6322 chars)"
 }
 ],
 "temperature": 1,
 "presence_penalty": 0,
 "frequency_penalty": 0
}

{
 "id": "chatcmpl-0a38a5eb-da8b-4485-92e7-91eef8955916",
 "model": "llama3.2:latest",
 "usage": {
 "total_tokens": 2026,
 "prompt_tokens": 2008,
 "completion_tokens": 18,
 "prompt_tokens_details": null,
 "completion_tokens_details": {
 "text_tokens": null,
 "audio_tokens": null,
 "reasoning_tokens": 0,
 "accepted_prediction_tokens": null,
 "rejected_prediction_tokens": null
 }
 },
 "object": "chat.completion",
 "choices": [
 {
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "{\"name\": \"HassBroadcast\", \"arguments\": {\"message\": \"Hello\"}}",
 "tool_calls": null,
 "function_call": null
 },
 "finish_reason": "stop"
 }
 ],
 "created": 1757222298,
 "system_fingerprint": null
}

https://github.com/BerriAI/litellm/issues/11273

https://github.com/BerriAI/litellm/blob/9a62b9bdb9ff217a0683047756588b2f5bd59c27/litellm/litellm_core_utils/prompt_templates/factory.py#L3842

Too much Vibe coding

Then I started digging into the code to understand why LiteLLM thinks this provider/model doesn’t support structured tool calls.

I came across this code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


if (
 custom_llm_provider == "ollama"
 and custom_llm_provider != "text-completion-openai"
 and custom_llm_provider != "azure"
 and custom_llm_provider != "vertex_ai"
 and custom_llm_provider != "anyscale"
 and custom_llm_provider != "together_ai"
 and custom_llm_provider != "groq"
 and custom_llm_provider != "nvidia_nim"
 and custom_llm_provider != "cerebras"
 and custom_llm_provider != "xai"
 and custom_llm_provider != "ai21_chat"
 and custom_llm_provider != "volcengine"
 and custom_llm_provider != "deepseek"
 and custom_llm_provider != "codestral"
 and custom_llm_provider != "mistral"
 and custom_llm_provider != "anthropic"
 and custom_llm_provider != "cohere_chat"
 and custom_llm_provider != "cohere"
 and custom_llm_provider != "bedrock"
 and custom_llm_provider != "ollama_chat"
 and custom_llm_provider != "openrouter"
 and custom_llm_provider != "vercel_ai_gateway"
 and custom_llm_provider != "nebius"
 and custom_llm_provider not in litellm.openai_compatible_providers
):
 if custom_llm_provider == "ollama":
 # X
 elif (
 # Y
 else:
 # Z

What is going on here?

Let’s just use some basic boolean algebra. If A AND NOT X AND NOT Y AND NOT Z simplifies to If A and in the above code, the Y and Z code paths are not possible to hit. Thus, this code is equivalent to:

1
2


if custom_llm_provider == "ollama":
 # X

Somebody filed an issue asking about this, but the maintainers didn’t understand the problem and it auto closed. Yet people keep adding new blocks of code to this method.

LiteLLM crashes during Ollama

For the longest time, LiteLLM would just crash any time I tried to work with Ollama with an error: Unclosed client session. The issue just sat there https://github.com/BerriAI/litellm/issues/11657

Slow as molasses

I have no idea why LiteLLM was so slow for me randomly. I experienced this across multiple different versions including up to my latest tested version v1.81.3-stable. XHR requests would take up to 3 minutes! This was running on a node with plenty of CPU, RAM, against a Postgres database

It’s so slow, yet somehow I can end up with duplicate models because I don’t know the requests are actually succeeding.

It’s not running on a slow computer at all, it’s got 64GB of RAM, 8 cores, not overloaded.

Conclusion

I can’t take it anymore. I’m either an idiot or something is seriously broken. I don’t even know what stable means anymore. I’m building my own LLM Proxy. It won’t have all the features, but at least the proxy will work. Stay tuned for a post.

Ads and data brokers are out of control

Sun, 01 Mar 2026 10:00:00 -0800

Digital advertising is everywhere nowadays. However, they are actually a giant risk to privacy and now, safety. To be successful, digital advertising depends on showing you highly targeted advertisements, which ultimately incentivizes them to build up profiles about you via your browsing history, search queries, location, demographics, and even behavioral patterns. More data about you means they can find ads that you’re more likely to be influenced by.

Data is collected via cookies and tracking pixels, which are invisible images that websites include to correlate an ad click with a user actually buying something. For example, in my post about Monarch Money and ad networks, the company embeds tracking images from TikTok and Reddit in their secure portal meaning that TikTok would know that the given user cares about finances. Now they can target usersr with more relevant ads.

Now you go to a restaurant and store that ask you if you want to sign up for a rewards program. If you come back 6 times, your 7th coffee is free! What a deal. Just enter your phone number and you’re in. Well, usually those programs are implemented by tech companies like Square point-of-sale systems who can then aggregate and use that to increase their surveillance.

What are data brokers?

Data brokers are the name for a company which makes money by finding information about people from as many different sources they can, aggregating it up to a specific person, then selling that package of information to other companies. For example, a data broker could scrape your local county’s property records which contains addresses, owner names, sale dates, and property taxes. That starts to associate names with income and financial statuses.

Court records can give a list of parking tickets and other information that’s relevant for background checks, a car insurance company looking to see how many tickets you have, or a prospective employer rescinding an offer.

Data breaches are hitting every company now accidentally leaking your personal information to the dark web because there’s no financial penalties for it. Nothing stops a data broker from finding these dumps and adding it to their package of information about you.

Where does the government come in?

The United States’ fourth amendment says:

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

This is generally interpreted to mean that the government has to have a warrant to “search” and monitor citizens and hopefully

However, if a company comes along and sells information about people, it’s not considered a search. The government can go to a data broker and say I’d like to buy everything you have and now they can do anything with that data. A hostile government could use that to track down people that disagree with them.

Block Ads

Why?

the FBI recommends it
ads are used as part of data brokers to build up shadow profiles
the government can buy access to this

Easy - Use Firefox with uBlock Origin If you’re using Google Chrome, I recommend switching to Mozilla Firefox because Google Chrome syncs your browsing history to their servers Evident here. This data can be used to build

Easy - Disable advertising identifier on iOS and Android Android and iOS phones have a built in advertising ID that is unique to your phone that doesn’t change. This identifier is used by ad networks to uniquely associate data to your phone, which by extension, provides information on you. Regularly rotating and/or deleting this id means that those ad networks can’t build this profile.

This will mean that advertisements will become less personalized to you, but this is good. On Android, go to Settings, search for advertising id, To do this, follow the steps here

Medium - Use DNS-based ad blocking Why? DNS-based ad-blocking is complementary to browser ad-block extensions. When setup, it can block some (but not all) ads in other apps on your phone and across your entire network.

How? On your phone? Try using NextDNS. Setup an account and enable any privacy filters you want. I like:

Blocklists - NextDNS Ads & Trackers Blacklist
Native Tracking Protection
Block Disguised Third Party Trackers

Hard - Self-host a DNS-based ad-blocker If you’re technically skilled, then you should consider running a Pi-Hole or Adguard DNS server at your home network. You’ll need a computer that always runs, such as a raspberry pi or other efficient computer, and will configure your DHCP server to hand out your DNS filter computer to all devices on the network. For more info, see the Adguard guide.

Data Brokers

What can you do about data brokers? The US doesn’t have a nation-wide privacy law, like GDPR, to control how companies store or process your data. Your only option is to opt-out or request deletion of your data to ask them to delete what they know about you. Unfortunately, there are a LOT of data brokers and it’s impractical to be able to find every one and opt out. Targeting the biggest ones is better than nothing

Of course, capitalism means you can pay somebody to remove yourself from data brokers. I personally use DeleteMe (or non-affiliate.)

https://www.consumerreports.org/electronics/personal-information/services-that-delete-data-from-people-search-sites-review-a2705843415/

Unfortunately, opting out of data brokers is not a big win because eventually your data will come back and you have to opt out again.

Some states in the US have privacy laws like GDPR for residents of those states. For example, in California, you can request deletion of your data from many brokers all at once at the CA DROP site. See more states here.

Conclusion

In conclusion, the unchecked power of digital advertising and data brokers poses a threat to our privacy and safety. By collecting vast amounts of personal data, these entities create detailed profiles that can be exploited not only for targeted ads but also by malicious actors or even hostile governments. The ability of data brokers to aggregate and sell sensitive information, often without meaningful regulation, works around limitations on the US Constitution to undermine rights.

While opting out of data brokers and using tools like ad blockers can help mitigate some of these risks, the problem is systemic. Until stronger privacy laws are enacted and enforced, you must remain vigilant and proactive in safeguarding their personal information. By taking steps to block ads, disable tracking identifiers, and remove your data from broker databases, you can reclaim some control over your digital life. Privacy is not just a right—it’s a necessity in our increasingly data-driven world.

References

Replatforming RKE1 to Nix-based K8s - Part 1

Sun, 08 Feb 2026 17:49:00 -0800

RKE1 (Rancher Kubernetes Engine 1) was Rancher’s first way of automatically deploying Kubernetes to a cluster. Think of it like minikube or on-prem EKS or K3s. Three years ago, it was marked as end of life (EoL) with the last release being July 2025. They have no migration guide and their strategy is just rebuild the cluster. I have 3 nodes a bunch of services running in Kubernetes. I don’t want to take everything down and rebuild it all. Let’s rebuild it while the plane is in the air.

In this post, I’m going to take an existing NixOS worker node that used RKE1 to deploy the control plane and worker and recreate it entirely in-place with etcd with minimal

NixOS is a a declarative, reproducible operating system

I was already using NixOS to declaratively define 1 out of my 3 worker’s operating system and I wanted to see how it works at configuring Kubernetes. I had two still using Ubuntu Server and I wanted to consolidate into a single approach. I had my issues with Nix, but for servers once it’s done, it works reasonably well.

First Cut

flake.nix:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{
 description = "Nixos config flake";

 inputs = {
 nixpkgs.url = "github:nixos/nixpkgs";
 nixos-hardware.url = "github:NixOS/nixos-hardware/master";
 disko = {
 url = "github:nix-community/disko";
 inputs.nixpkgs.follows = "nixpkgs";
 };
 };

 outputs = { self, nixpkgs, disko, ... }@inputs: {
 nixosConfigurations.srv7 = nixpkgs.lib.nixosSystem {
 specialArgs = {inherit inputs;};
 modules = [
 # ... 
 ./parts/kubernetes.nix
 ];
 };
 };
}

kubernetes.nix:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110


{ config, lib, pkgs, nixpkgs-k8s, ... }:
let
 kubeMasterHostname = "localhost";
 kubeMasterAPIServerPort = 6443;
 clusterIpv4 = { "srv5" = "51.81.64.31"; };
 hostIpv4 = clusterIpv4."${config.networking.hostName}";
 hostIpv6 = { "srv5" = "2604:2dc0:100:1be8:beef:beef:beef:beef"; }."${config.networking.hostName}";
 sslBasePath = "/etc/kubernetes/ssl/";
 ipWithHyphens = builtins.replaceStrings ["."] ["-"] hostIpv4;
in
{
 services.etcd = {
 advertiseClientUrls = ["https://${hostIpv4}:2379"];
 listenClientUrls = ["https://0.0.0.0:2379" ];
 listenPeerUrls = ["https://0.0.0.0:2380" ];
 initialAdvertisePeerUrls = ["https://${hostIpv4}:2380"];
 certFile = "${sslBasePath}kube-etcd-${ipWithHyphens}.pem";
 initialClusterToken = "etcd-cluster-1";
 peerClientCertAuth = true;
 clientCertAuth = true;
 initialCluster = (lib.mapAttrsToList (n: v: "etcd-${n}=https://${v}:2380") clusterIpv4);
 keyFile = "${sslBasePath}kube-etcd-${ipWithHyphens}-key.pem";
 name = "etcd-${config.networking.hostName}";
 openFirewall = true;
 trustedCaFile = "${sslBasePath}kube-ca.pem";
 enable = true;
 };

 environment.systemPackages = with pkgs; [
 pkgs.cri-tools
 ];

 services.kubernetes = {
 package = pkgs.kubernetes;
 masterAddress = kubeMasterHostname;
 apiserverAddress = "https://${hostIpv4}:${toString kubeMasterAPIServerPort}";
 apiserver = {
 advertiseAddress = hostIpv4;
 allowPrivileged = true;
 apiAudiences = "unknown";
 enable = true;
 etcd = {
 servers = [
 # Only connect to local etcd to avoid cross node calls
 "https://${hostIpv4}:2379"
 ];
 keyFile = "${sslBasePath}kube-node-key.pem";
 certFile = "${sslBasePath}kube-node.pem";
 caFile = "${sslBasePath}kube-ca.pem";
 };
 extraOpts = "--etcd-prefix=/registry --service-account-lookup=true --anonymous-auth=false --service-node-port-range=30000-32767";
 kubeletClientCertFile = "${sslBasePath}kube-apiserver.pem";
 kubeletClientKeyFile = "${sslBasePath}kube-apiserver-key.pem";
 proxyClientCertFile = lib.mkForce "${sslBasePath}kube-apiserver-proxy-client.pem";
 proxyClientKeyFile = lib.mkForce "${sslBasePath}kube-apiserver-proxy-client-key.pem";
 securePort = kubeMasterAPIServerPort;
 serviceAccountKeyFile = lib.mkForce "${sslBasePath}kube-service-account-token-key.pem";
 serviceAccountIssuer = "rke";
 serviceAccountSigningKeyFile = lib.mkForce "${sslBasePath}kube-service-account-token-key.pem";
 serviceClusterIpRange = "10.43.0.0/16";
 tlsCertFile = lib.mkForce "${sslBasePath}kube-apiserver.pem";
 tlsKeyFile = lib.mkForce "${sslBasePath}kube-apiserver-key.pem";
 };
 scheduler = {
 enable = true;
 kubeconfig = {
 certFile = "${sslBasePath}kube-scheduler.pem";
 keyFile = "${sslBasePath}kube-scheduler-key.pem";
 };
 };
 controllerManager = {
 enable = true;
 extraOpts = "--service-cluster-ip-range=10.43.0.0/16";
 serviceAccountKeyFile = lib.mkForce "${sslBasePath}kube-service-account-token-key.pem";
 kubeconfig = {
 certFile = "${sslBasePath}kube-controller-manager.pem";
 keyFile = "${sslBasePath}kube-controller-manager-key.pem";
 };
 };
 pki.enable = false; # Skip Nix's cert generation. We'll use our own. We still need to handle rotation
 flannel.enable = false;
 addonManager.enable = false;
 caFile = lib.mkForce "${sslBasePath}kube-ca.pem";
 clusterCidr = "10.42.0.0/16";

 proxy = {
 enable = true;
 kubeconfig = {
 caFile = "${sslBasePath}kube-ca.pem";
 certFile = "${sslBasePath}kube-proxy.pem";
 keyFile = "${sslBasePath}kube-proxy-key.pem";
 };
 };

 kubelet = {
 clusterDns = ["10.43.0.10"];
 cni.packages = [];
 enable = true;
 hostname = config.networking.hostName;
 kubeconfig = {
 keyFile = "${sslBasePath}kube-node-key.pem";
 certFile = "${sslBasePath}kube-node.pem";
 };
 tlsCertFile = "${sslBasePath}kube-node.pem";
 tlsKeyFile = "${sslBasePath}kube-node-key.pem";
 nodeIp = "${hostIpv4},${hostIpv6}";
 extraOpts = "--kube-reserved=cpu=1000m,memory=512Mi --root-dir=/var/lib/kubelet";
 };
 };
}

SSL

First problem is trivial to fix because the apiserver runs as user kubernetes:

journalctl -u kube-apiserver -f

1
2
3
4


options.go:249] external host was not specified, using 51.81.64.31
run.go:72] "command failed" err="failed to parse service-account-issuer-key-file: open /etc/kubernetes/ssl/kube-service-account-token-key.pem: permission denied"
Main process exited, code=exited, status=1/FAILURE
Failed with result 'exit-code'.

To fix it, let’s set the permissions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[nix-shell:/etc/nixos]# systemctl show kube-apiserver | grep User
User=kubernetes
DynamicUser=no
PrivateUsers=no
PrivateUsersEx=no

[nix-shell:/etc/kubernetes/ssl]# ls -la
total 128
drwxr-xr-x 1 root root 1696 Mar 6 2025 .
drwxr-xr-x 1 root root 68 Sep 17 20:17 ..
-rw------- 1 root root 1675 Sep 4 2024 kube-apiserver-key.pem
-rw------- 1 root root 1330 Sep 4 2024 kube-apiserver.pem
-rw------- 1 root root 1679 Sep 4 2024 kube-apiserver-proxy-client-key.pem
-rw------- 1 root root 1151 Sep 4 2024 kube-apiserver-proxy-client.pem
-rw------- 1 root root 1679 Sep 4 2024 kube-apiserver-requestheader-ca-key.pem
-rw------- 1 root root 1123 Sep 4 2024 kube-apiserver-requestheader-ca.pem
-rw------- 1 root root 1675 Sep 4 2024 kube-ca-key.pem
-rw------- 1 root root 1058 Sep 4 2024 kube-ca.pem
...

Fix it by updating permissions:

1
2
3


chown kubernetes:kubernetes /etc/kubernetes/ssl/*.pem
chmod o+r /etc/kubernetes/ssl/kube-ca.pem
chown etcd /etc/kubernetes/ssl/kube-etcd-*.pem

Pause image

Kubernetes and containerd depend on something called a pause image which is like a Docker image that contains a single process that is used to setup resource namespaces. In my opinion, it’s a leaky abstraction that was first introduced due to Docker’s limitations, then carried forward into containerd, but shouldn’t have been exposed.

1

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "736e57677163ae0948aaa7425f9dee4776a84ac65eeab146b4303ab60820b990": failed to get sandbox image "pause:latest": failed to pull image "pause:latest": failed to pull and unpack image "docker.io/library/pause:latest": failed to resolve image: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Startup depends on a pause image, but the pause image can get garbage collected. Let’s see what’s going on.

[nix-shell:/etc/nixos]# cat /nix/store/0k19qdzb5vygrg8s6d0rvjghl1p0qjqa-containerd-config-checked.toml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


oom_score = 0
root = "/var/lib/containerd"
state = "/run/containerd"
version = 2

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "pause:latest"

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
max_conf_num = 0

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd

Now how do I pin the images? Several issues pointed the blame in different ways.

I ended up with this work around:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{ config, lib, pkgs, ... }:
let
 # ...
 infraContainer = pkgs.dockerTools.buildImage {
 name = "pause";
 tag = "latest";
 copyToRoot = pkgs.buildEnv {
 name = "image-root";
 pathsToLink = [ "/bin" ];
 paths = [ pkgs.kubernetes.pause ];
 };
 config.Cmd = [ "/bin/pause" ];
 };
in
{
 systemd.services.kubelet = {
 preStart = lib.mkForce ''
 set -e
 ${pkgs.containerd}/bin/ctr -n k8s.io image import --label io.cri-containerd.pinned=pinned ${infraContainer}
 '';
 };
}

However, after finishing my cluster, I came across this Nixpkgs commit that fixes the issue, but doesn’t work in my case because of the next issue.

Dude, where’s my CNI?

1

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "65949aeccfed7a3571f721c79e1d5e02d02161a3a3efa343936b34a0480a1f2b": plugin type="calico" failed (add): failed to find plugin "calico" in path [/opt/cni/bin]

The next problem is caused by the Calico CNI driver getting deleted when kubelet restarts. The calico-driver-deployer DaemonSet is responsible for deploying it, but Nixpkgs has this line that deletes all CNI binaries. Intended to be reproducible, it breaks any custom CNI deployments.

1
2
3
4
5
6
7


preStart = ''
 // ...
 rm /opt/cni/bin/* || true
 ${concatMapStrings (package: ''
 echo "Linking cni package: ${package}"
 ln -fs ${package}/bin/* /opt/cni/bin
 '') cfg.cni.packages}

Thus, I have to disable this preStart code.

kubernetes.nix:

1
2
3
4
5
6


{
 systemd.services.kubelet.preStart = lib.mkForce ''
 mkdir -p /opt/cni/bin/
 ${pkgs.containerd}/bin/ctr -n k8s.io image import --label io.cri-containerd.pinned=pinned ${infraContainer}
 '';
}

Why’s my service cidr broken?

This one was very strange. I saw some errors in Kubernetes:

1
2
3
4
5
6


kind: Event
message: >-
 Cluster IP [IPv4]: 10.43.27.126 is not within any configured Service CIDR;
 please recreate service
series:
 count: 2194

Based on my reading, I think I accidentally swapped my serviceClusterIpRange and clusterIpRange in the configuration at some point and it got stuck in my cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


journalctl -u kube-apiserver

Oct 03 22:28:21 srv5 kube-apiserver[825540]: I1003 22:28:21.386392  825540 cidrallocator.go:301] created ClusterIP allocator for Service CIDR 10.0.0.0/24
Oct 03 22:28:21 srv5 kube-apiserver[825540]: I1003 22:28:21.333430  825540 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
Oct 03 22:28:21 srv5 kube-apiserver[825540]: I1003 22:28:21.386392  825540 cidrallocator.go:301] created ClusterIP allocator for Service CIDR 10.0.0.0/24
Oct 03 22:28:21 srv5 kube-apiserver[825540]: I1003 22:28:21.433485  825540 cache.go:39] Caches are synced for APIServiceRegistrationController controller
Oct 03 22:28:21 srv5 kube-apiserver[825540]: I1003 22:28:21.444773  825540 default_servicecidr_controller.go:227] inconsistent ServiceCIDR status, global configura
tion: [10.0.0.0/24] local configuration: [10.43.0.0/16], configure the flags to match current ServiceCIDR or manually delete the default ServiceCIDR
Oct 03 22:28:21 srv5 kube-apiserver[825540]: I1003 22:28:21.444968  825540 event.go:389] "Event occurred" object="kubernetes" fieldPath="" kind="ServiceCIDR" apiVe
rsion="networking.k8s.io/v1" type="Warning" reason="KubernetesDefaultServiceCIDRInconsistent" message="The default ServiceCIDR [10.0.0.0/24] does not match the fla
g configurations [10.43.0.0/16]"
Oct 03 22:28:21 srv5 kube-apiserver[825540]: E1003 22:28:21.535706  825540 repairip.go:353] "Unhandled Error" err="the ClusterIP [IPv4]: 10.43.28.139 for Service c
attle-system/rancher-webhook is not within any service CIDR; please recreate" logger="UnhandledError"
Oct 03 22:28:21 srv5 kube-apiserver[825540]: E1003 22:28:21.537293  825540 repairip.go:353] "Unhandled Error" err="the ClusterIP [IPv4]: 10.43.192.190 for Service datastore/mysql-lb is not within any service CIDR; please recreate" logger="UnhandledError"
...

Searching online found a few issues, but the solution wasn’t obvious:

There was a hidden resource servicecidrs/kubernetes that kubectl was unable to modify. The only way to fix it was actually to edit the backing store for Kubernetes, in etcd.

Looking in etcd, I found the culprit. Note content is stored in binary.

1
2
3
4
5
6
7


docker exec -ti etcd etcdctl get /registry/servicecidrs/kubernetes
/registry/servicecidrs/kubernetes
....

10.0.0.0/24▒?
=
ReadyTrue����*2 Kubernetes Service CIDR is ready▒"

To fix it, run the following commands.

1
2
3
4
5


docker exec -ti etcd etcdctl get /registry/servicecidrs/kubernetes > backup

kubectl delete service kubernetes

docker exec -ti etcd etcdctl del /registry/servicecidrs/kubernetes

Then restart kube-apiserver. It’ll auto auto recreate it and it’s fixed. However, don’t be like me and just configure your Nix flake correctly the first time.

Pin Kubernetes

One of my problems with Nix is that when you use Nixpkgs, you almost always end up using the latest version of every package.

1
2
3


{
 services.kubernetes.apiserver.enable = true;
}

Doing the above means one day you’re using Kubernetes 1.33.4, and the next day you’re on 1.34.0. Kubernetes upgrades aren’t safe to be done blindly. Nodes need to updated after the control plane and features get deprecated. I don’t mind automatically performing minor updates like 1.33.3 to 1.33.4, but major updates need to be opt-in.

We can pin to a specific version like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


{ config, lib, pkgs, ... }:
let

 package = pkgs.kubernetes.overrideAttrs (oldAttrs: rec {
 version = "1.34.2";

 src = pkgs.fetchFromGitHub {
 owner = "kubernetes";
 repo = "kubernetes";
 rev = "v${version}";
 sha256 = "3rQyoGt9zTeF8+PIhA5p+hHY1V5O8CawvKWscf/r9RM=";
 };
 });
in
{
 services.kubernetes.package = package;
}

This isn’t yet perfect because it doesn’t ensure that the api server is updated first across the cluster as per the official docs, but it’s a first step.

Conclusion

In this post, I walked through how to take a Kubernetes node running RKE1 and deploy Kubernetes via Nix+Nixpkgs instead of RKE1. In my next post, I’ll show to swap a Ubuntu server to NixOS using nixos-anywhere

Syncing my Christmas lights to my Sonos

Tue, 30 Dec 2025 00:00:00 +0000

Last year, I setup a Christmas lights show at my house. I started with some basic light sequences just to learn. I wrote a post on the basics. This year, I upped the ante and added more lights and starting making sequences linked to music.

I have one light controller running Falcon Controller/FPP, a Kulp K8-B controller. How do I get sound out? I looked at options for getting sound out to a speaker. For my first pass, I decided to push it to a Sonos Move speaker since I had one.

How do I connect a speaker?

My Kulp K8-B controller is a hat that sits on top of a BeagleBoard Black.

The BeagleBoard, like a Raspberry Pi, runs Linux. It has no 3.5mm audio output, but has USB and Ethernet. I could attach a USB-attached audio player like this, but then I’d have to drill a hole in my weather enclosure and not ruin that. Most people run them to speakers and/or FM transmitters for cars, but I wanted to be able to play it without being in a car. Instead, since I already several Sonos speakers including a Move that can be taken outside. The Sonos speakers inside the house could be synchronized to the lights I had inside for parties.

Control Protocols

I have two different systems, Sonos and FPP. Neither of them talk to each other, so I needed to first understand how they independently work.

FPP has two different protocols that can be used:

Pixel control data (e.g. DDP an E1.31) - Sends individual pixel data to a receiver. Pixel data has to be sent at 20hz - 40hz to match the sequence
Timing data - FPP MultiSync- Sends timing and song names over the network and the player loads a locally saved sequence. Timing data is sent every regularly around every 100ms-1s and players internally speed up or slow down to match the ticks.

The FPP MultiSync protocol sounds like it would be best to integrate with since I can send period sync packets and the player handles the pixel data.

Wireshark Dissector

The first thing I did was write a wireshark packet dissector to help me understand what the packets look like and construct my own packets. There’s no built-in dissector in Wireshark for FPP MultiSync or DDP, so packets look like an opaque blob of bytes:

The protocol is defined in this spec. Wireshark exposes a Lua based Dissector API. Using this, I wrote a simple program that looked at packets and extracted out the relevant bytes into structured Wireshark fields:

The code can be found here. To load it into Wireshark, go to Help → About → Folders. Click on Personal Lua Plugins and paste the files into that folder.

The Control Script

I can either listen to another player send them or try to send them myself. If I listen to the packets, then I’d have to speed up or slow down the Sonos player. Given this forum post on how the players internally work and the fact that it’s easier for our ears to hear discontinuities in the audio, I let the speakers play, then send FPP MultiSync packets every second to a multicast address that all players subscribe to.

The source code can be found here: https://github.com/ajacques/sonos-fpp-sync

A high-level explanation on how it works: Upon startup, it loads a playlist of songs and fseq files (generated by xLights), then loads the songs into the Sonos queue. Songs are served by a local HTTP web server running from the script. I considered streaming from a Jellyfin server, but I couldn’t solve some network access issues, and get Sonos to stream with the right codec and access tokens.

Once the songs are loaded, it subscribes to media events from Sonos using python-soco running using Python Twisted. Twisted is an event-based networking engine for Python that handles the async networking required for Sonos subscriptions, responding to HTTP requests to stream media, and sending datagrams.

When the Sonos speaker starts playing a song, it sends a message to my script saying what song is playing. I then tell the FPP players to load the sequence containing the pixel data and prepare to start playing. I then start a timer that runs every second that asks Sonos how many seconds have elapsed in the song. I use that to derive the frame counter (# of seconds elapsed / # of seconds in song * number of frames in fseq file) and send a sync packet to the FPP players.

There is a problem here. Sonos gives me elapsed time with a resolution of 1 second, combined with the round-trip time of ~50ms, that means the pixels can be off beat. I’ve noticed this when playing some sequences. I don’t have a fix for this yet, but had the idea to add some jitter to tick timer and average out the calculated start times. That’s a problem for next year.

I’d put a video, but I don’t want a DMCA notice.

Source Code: https://github.com/ajacques/sonos-fpp-sync

More infrastructure doesn't fix using the wrong infrastructure

Tue, 16 Dec 2025 00:00:00 +0000

This is a continuation of my previous post where I talked about the challenges of using serverless/Function as a Service (FaaS) compute systems for ETL (Extract, Transform, Load) jobs. It sat in my drafts folder for a long time, so I just decided to publish it as is.

I used to work at AWS, and predictably we used a lot of AWS cloud services. In many cases, when an engineer looks for a service platform, they’ll often go directly to AWS Lambda because “it’s Serverless” with the justification that it’s simple and alternatives are too complex and not worth considering.

In this post, I introduce another “heavily inspired by true events” problem space in which Serverless was the wrong choice. A queue processing system in AWS Lambda seems like a class example of using Serverless. Let’s walk through the problem.

Queue Processing Lambda

A pattern I saw was using AWS Lambda as a queue processing feeding into another system with a limited throughput. The target service(s) had different constraints, such as a maximum number of tps (transactions per second) or a maximum number of in-flight operations before it would reject requests.

Diagram Here - Maybe

In a naïve implementation (I saw this in several systems), SQS would feed into the Lambda function, which then would call the service. If it throttles, retry until it succeeds.

Do you see any problems with this approach? Hopefully a few bells will be going off.

If you get a throttle, how many times do you retry? How quickly should you retry?
What happens if the number of retries exceeds the Lambda function timeout and it gets cut off in the middle of processing?
Is the batch size big or small? If it’s too big, will it exceed the short-term throttling window?
What happens if Lambda starts scaling out horizontally because you have more inbound messages or because of the retry delays are starting to take more time so it thinks it needs more functions?
What happens if the throttle period means you can’t do any work for several minutes? Lambda will continue to deliver messages into your function even if you know you can’t do anything causing failed deliveries and presumably you have a DLQ configured (you do don’t you?)
Is the process idempotent?

Let’s take an example service with a limit of 5 tps, each request kicks off a process that runs for 5-20 minutes, and there’s a maximum number of 10 workflows allowed for this client. If you didn’t think about this and used the defaults, then you’ll get 10 messages in that batch. Then pass those to the API sequentially and the API responds in <100ms, then on the 6th request, get throttled. Now depending on the periodicity of the throttling algorithm, it could clear up immediately or take a few seconds. Not terrible.

Let’s make it worse. Say you’ve got huge queue of messages. AWS Lambda automatically scales out your number of invocations from 1 to 10 concurrent invocations. Now, you’re trying to do 10 x (10 / 100ms) = 50 requests per second with 90% of them getting throttled.

Reduce concurrency

Is this fixable without moving from AWS Lambda? What knobs do we have? Well, we could limit the number of horizontal function invocations to 1 which would constrain the maximum number of tps from:

5 (event source mapping batch) \times 1000 invocations \times \frac{1 second}{100ms average latency} = 50,000 tps

to:

1 - 5 (event source mapping batch) \times 1 invocations \times \frac{1 second}{100ms average latency} = 50 tps

That reduces the worst case quite a bit, but it still exceeds my hypothetical example.

Taking control of the process life-cycle

To break free from this problem, we have to take control of the entire control loop and decide when to fetch messages, how many messages to fetch, and how to react to errors. Not to worry, this is easier than it sounds.

The process starts to look something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


void main(String[] args) {
 int numMessagesInBatch = 5;
 while (true) {
 List<Message> messages = amazonSQSClient.receiveMessages(numMessagesInBatch)

 try {
 for (message : messages) {
 // Process each message
 }
 } catch (ThrottleException e) {
 // Ack any successful messages
 // Sleep and optionally reduce numMessagesInBatch
 } catch (RuntimeException e) {
 // Handle other errors as appropriate
 }
 }
}

Conclusion

Picking the simple solution or using a technology that you’re already familiar with can sometimes be the pragmatic choice.

Picking the right tool for the job is important and sometimes what appears to be a simple solution actually ends up being more complicated when you factor in all failure modes. Serverless is great for a class of problems. If you have a service API that’s infrequently called, the above problems don’t appear because failures apply back-pressure to the caller to slow down and retry. If you’re processing a queue and you’re likely to cause throttling while processing, Lambda and FaaS-based systems can fall-over. Don’t forget to leverage big data compute specialized systems, like Glue, EMR, and even Batch if you have that situation.

Blogging on the Fediverse with ActivityPub

Mon, 15 Dec 2025 00:00:00 +0000

Previously, if you wanted to subscribe to changes from this blog, you’d have to subscribe to the RSS feed, but as of today you can also subscribe to it in your preferred Fediverse client, like Mastodon. Note this is considered Beta quality. If you have any issues, let me know.

What is the Fediverse? It’s a protocol for federated (meaning many independently operated) social networks, kind of like email. Under the hood, it uses a protocol called ActivityPub to define the interactions between different servers.

There’s a number of big implementations of this, like Mastodon, that I could have used. However, I wanted to see if it was possible to integrate directly into my static website generator, Hugo and generate all the content directly out of the posts I already write without having to maintain another program and expose another domain name for people to remember (e.g. blog@mastodon.technowizardry.net.)

This post walks through the work I did to make this work.

What is ActivityPub?

ActivityPub is an open standard protocol designed to enable decentralized social networking across different platforms and services. Published by the W3C organization, the same org that publishes the HTML standard, it allows users to maintain their online identities and connections while moving between different websites, apps, and services without losing access to their content or network.

By defining an interoperable protocol, different software, like Mastodon, Lemmy, Pleroma can all subscribe and publish notes to different instances and software. The federation part means that you can subscribe to notes on a different server. My goal is to have readers be able to subscribe to posts in Mastodon and eventually be able to like and even comment on them.

The WebFinger

The first thing that happens when you follow somebody else is your software issues a “webfinger” request (spec). The client will GET https://www.technowizardry.net/.well-known/webfinger?resource=acct:blog@technowizardry.net. The response tells clients that I do support ActivityPub and the response looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
 "subject": "acct:blog@technowizardry.net",
 "aliases": [
 "https://www.technowizardry.net/",
 "https://www.technowizardry.net/author/adam"
 ],
 "links": [
 {
 "rel": "self",
 "type": "application/activity+json",
 "href": "https://www.technowizardry.net/author/adam"
 }
 ]
}

To generate this with Hugo, create a file layouts/index.webfinger.ajson:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
 "subject": "acct:blog@{{ strings.TrimRight "/" (replace $.Site.BaseURL "https://" "") }}",
 "aliases": [
 "{{ $.Site.BaseURL}}",
 "{{ $.Site.BaseURL }}author/adam"
 ],
 "links": [
 {
 "rel": "self",
 "type": "application/activity+json",
 "href": "{{ $.Site.BaseURL }}author/adam"
 }
 ]
}

The subject needs to match the value passed by the client in the /webfinger?resource=acct:blog@technowizardry.net. Since I only support a single account, I can hard-code the value. The links array will tell clients where to find my user details. This is critical for ActivityPub.

Next we need to tell Hugo to generate this file (set in config.yaml). Note that I’m going to generate the files with an .ajson extension to distinguish between this and the HTML outputs. More on that later.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


mediaTypes:
 # ...
 # The template file will have the extension .ajson
 application/jrd+json:
 suffixes:
 - ajson

outputFormats:
 # ...
 WEBFINGER:
 mediaType: application/jrd+json
 notAlternative: true
 # Hugo will output to public/.well-known/webfinger/index.ajson
 path: .well-known/webfinger

outputs:
 home:
 # ...
 - WEBFINGER

Generating the outbox

The ActivityPub outbox contains a historical listing of ActivityPub events (my blog posts) that other servers can download to catch-up on old events. Unfortunately, it doesn’t seem to be well used. For example, Mastodon does not pull old posts.

I’m going to generate it anyway just to be safe in-case there is a server that does support it. The template iterates through every published post and emits a single JSON object for every post, much like the RSS feed is generated.

Note the way that I serialize the content and summary fields. While researching this post and implementing it, I found several other implementations that incorrectly serialized the objects where HTML entities would be included in the post or they would even generate invalid JSON objects. For more information, see my other post on fixing common Hugo encoding problems.

layouts/index.activitypub_outbox.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


{{- $pctx := . -}}
{{- if .IsHome -}}{{ $pctx = .Site }}{{- end -}}
{{- $pages := slice -}}
{{- if or $.IsHome $.IsSection -}}
{{- $pages = .Site.RegularPages -}}
{{- else -}}
{{- $pages = $pctx.Pages -}}
{{- end -}}
{{- $pages := where $pages "Params.hidden" "!=" true -}}
{{- $limit := .Site.Config.Services.RSS.Limit -}}
{{- if ge $limit 1 -}}
{{- $pages = $pages | first $limit -}}
{{- end -}}
{
 "@context": "https://www.w3.org/ns/activitystreams",
 "id": "{{ $.Site.BaseURL }}activitypub/outbox",
 "summary": "{{ $.Site.Title }}",
 "type": "OrderedCollection",
 {{- $notdrafts := where $pages ".Draft" "!=" true }}
 {{- $all := where $notdrafts "Type" "in" (slice "posts")}}
 "totalItems": {{ len $all }},
 "orderedItems": [
 {{- range $index, $element := $all }}
 {{- if ne $index 0 }}, {{ end }}
 {
 "@context": "https://www.w3.org/ns/activitystreams",
 "id": "{{.Permalink}}-create",
 "type": "Create",
 "actor": "{{ $.Site.BaseURL }}author/adam",
 "object": {
 "id": "{{ .Permalink }}",
 "type": "Article",
 "content": {{ .Content | htmlUnescape | jsonify (dict "noHTMLEscape" true) }},
 "url": {{ .Permalink | jsonify }},
 "summary": {{ printf "%s<br><br>%s" .Title .Summary | jsonify }},
 "attributedTo": "{{ $.Site.BaseURL }}author/adam",
 "to": "https://www.w3.org/ns/activitystreams#Public",
 "published": {{ dateFormat "2006-01-02T15:04:05-07:00" .Date | jsonify }}
 }
 }
 {{- end }}
 ]
}

Time to update the config.yaml again:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


mediaTypes:
 application/activity+json:
 suffixes:
 - ajson

outputFormats:
 ACTIVITY_OUTBOX:
 mediaType: application/activity+json
 notAlternative: true
 baseName: outbox

outputs:
 home:
 - HTML
 - ACTIVITY_OUTBOX

Generate per-post files

Even though Mastodon doesn’t download old posts automatically, it can still open any post. By pasting the URL, and clicking “Open URL in Mastodon”, Mastodon will issue a GET request to that URL with the header Accept: application/activity+json expecting to download the post in ActivityPub JSON format.

Right now, it’s passing back as HTML. Let’s generate something per post. Again, update the config.yaml to generate this new file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


outputFormats:
 ACTIVITY_USER:
 mediaType: application/activity+jsonindex
 notAlternative: true
 baseName: activitypub
 POST_JSON:
 mediaType: application/activity+json
 notAlternative: true

outputs:
 page:
 - HTML
 - POST_JSON

Let’s refactor this and the outbox and create a new partial that is shared between the outbox and post. The partial has to have the extension .html because that’s how Hugo works.

layouts/partials/post_main_blob.html:

1
2
3
4
5
6
7
8


"id": "{{ .Permalink }}",
"type": "Article",
"content": {{ .Content | htmlUnescape | jsonify (dict "noHTMLEscape" true) }},
"url": {{ .Permalink | jsonify }},
"summary": {{ printf "%s<br><br>%s" .Title .Summary | jsonify }},
"attributedTo": "{{ $.Site.BaseURL }}author/adam",
"to": "https://www.w3.org/ns/activitystreams#Public",
"published": {{ dateFormat "2006-01-02T15:04:05-07:00" .Date | jsonify }}

layouts/posts/single.post_json.ajson:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


{
 "@context": [
 "https://www.w3.org/ns/activitystreams",
 {
 "ostatus": "http://ostatus.org#",
 "conversation": "ostatus:conversation",
 "sensitive": "as:sensitive"
 }
 ],
 {{ partial "post_main_blob" . }},
 "cc": [
 "{{ $.Site.BaseURL }}author/adam/followers"
 ],
 "sensitive": false,
 "attachment": [],
 "tag": [],
 "replies": {
 "id": {{ printf "%sreplies" .Permalink | jsonify }},
 "type": "Collection",
 "first": {
 "type": "CollectionPage",
 "next": {{ printf "%sreplies-page" .Permalink | jsonify }},
 "partOf": {{ printf "%sreplies" .Permalink | jsonify }},
 "items": []
 }
 }
}

Making the Accept header work

This is the part where your environment may look different than mine and may differ. For example, if you’re running on Vercel, then this approach would be better. If you were using Azure Websites, then this also works. Both of these generate static files that include references to small functions as a service to handle the inbox and followers lists. These can be hosted on any cloud provider.

Prior to this project, I used pure Hugo to generate static content and Mastodon was able to work with this to load posts. I built everything using a Hugo Docker image and packaged the content into an NGINX Docker container that was run on my Kubernetes cluster. However, this is not enough to implement ActivityPub.

First, we need to check the Accept header to see if the client is requesting HTML or if it’s requesting an ActivityPub JSON blob of the item.

Attempt 1 - Using NGINX

My first attempt was to implement this using NGINX’s configuration and it looked like the below. The following implemented a conditional based on the Accept header and returned the index.ajson file if the client passed in the special MIME type or returns index.html in any other case.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


map $http_accept $ap_suffix {
 default "/index.html";
 "~*application\/activity\+json" "/index.ajson";
}

server {
 listen 80;
 server_name localhost;
 root /usr/share/nginx/html;

 types {
 application/activity+json ajson;
 text/html html;
 text/css css;
 video/mp4 mp4;
 image/png png;
 }

 location / {
 try_files $uri $uri$ap_suffix =404;
 }

 location = /.well-known/webfinger {
 default_type application/jrd+json;
 try_files $uri/index.ajson =404;
 }

 location = /activitypub/outbox {
 try_files $uri$ap_suffix =404;
 }

 location = /activitypub/following {
 # $ap_suffix is either '.html' or '.ajson'
 try_files $uri$ap_suffix =404;
 }
}

This worked okay, but as I continued to implement this, I started to have issues. For example, when developing I had circular dependencies, and logic to implement request handling was getting split up into different Docker containers.

Next part of this series, I’ll show how I approached this and implemented a server-side follower store.

Conclusion

As you can see, this is one of the lengthier posts, for a seemingly “easy” function. This goes to show you that the simplist of tasks can take the most amount of time. But don’t worry, all you have to do as a user is to “like” and “subscribe” and donate to my coffee fund for more information like this.

To implement ActivityPub, we have to generate the webfinger file to allow clients to know where to go, an ActivityPub user page, an outbox to download all posts, and individual post files.

References

I like the idea of Nix, but don't enjoy using it

Fri, 28 Nov 2025 10:54:00 -0800

I’ve been playing with Nix and NixOS a lot more lately. I installed NixOS on one of my servers, I installed the Nix CLI on my laptop, I tried to use Nix to build a Docker image, I use Nix flakes.

This post was written from the perspective of a person new to Nix, but experienced with other computer languages. Thus, it’s probable that I might be doing something wrong or maybe complaining about something that’s obvious to you. However, these are issues that others may face.

It was also written over several months as I gathered issues, so even looking back, I see mistakes.

The Good Parts

The biggest advantage about Nix is that I can define an entire system using a text based language that I can check into a Git repo. The workflow of identifying a bug, making a change, running nixos-rebuild switch, verifying it works, then checking it in. Then six months from now being able to Git blame and see why I made a change is great.

The Bad Parts

Nix, the language

The Nix language is not-intuitive. I understand it’s not like most computing languages that are designed for performing operations and it’s intended to be a declarative configuration model, but as a developer with skills in other languages, I can’t

Variables

To declare a variable, you have to do this:

1
2
3
4


let
 x = 123
in
 blah

My problem is I might start with some code like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
 pkgs ? import <nixpkgs> {},
 content ? import ./website.nix {},
}:

{
 nginx = pkgs.nginxStable.overrideAttrs (oldAttrs: {
 # ...
 });

 nginxConfig = pkgs.writeText "nginx.conf" ''
 # blah
 '';
}

But if I want to refer to nginx in nginxConfig, I need to create a let block and define the variable away from the code that uses it, or create another let block that increases nesting.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


{
 pkgs ? import <nixpkgs> {},
 content ? import ./website.nix {},
}:

let
 nginx = pkgs.nginxStable.overrideAttrs (oldAttrs: {
 # ...
 });
{
 nginx = nginx;
 nginxConfig = pkgs.writeText "nginx.conf" ''
 # blah
 '';
}

This requires a change in the code to do something that is trivial in every other programming language.

Nix, the CLI

Errors

Having useful error messages is critical for developers, but nix does not give good error messages. Now, a lot of this is because I’m new to Nix and make lots of dumb mistakes when coding, but a language needs to be approachable by new people. Otherwise, they can’t turn into experienced devs.

Let’s see an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


error:
 … while calling the 'derivationStrict' builtin
 at <nix/derivation-internal.nix>:34:12:
 33|
 34| strict = derivationStrict drvAttrs;
 | ^
 35|

 … while evaluating derivation 'hugo-nginx-image.tar.gz'
 whose name attribute is located at /nix/store/vfvjcnwvwy1lrjfaz0wvvd5fwcf5gb4v-nixpkgs/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:336:7

 … while evaluating attribute 'buildCommand' of derivation 'hugo-nginx-image.tar.gz'
 at /nix/store/vfvjcnwvwy1lrjfaz0wvvd5fwcf5gb4v-nixpkgs/nixpkgs/pkgs/build-support/trivial-builders/default.nix:59:17:
 58| enableParallelBuilding = true;
 59| inherit buildCommand name;
 | ^
 60| passAsFile = [ "buildCommand" ]

 (stack trace truncated; use '--show-trace' to show the full, detailed trace)

 error: expected a set but found a function: «lambda @ ~/projects/technowizardry.net/nix/docker.nix:1:1»

A file that I wrote was only mentioned at the very last point. Let’s zoom in:

1
2
3
4
5
6
7
8
9


{
 pkgs ? import <nixpkgs> {},
 docker ? import ./nix/docker.nix,
 hugo ? import ./nix/hugo.nix {},
}:

pkgs.dockerTools.buildLayeredImage {
 # ...
}

Can you spot the mistake? I spent time hunting through the buildLayeredImage block, but that was entirely irrelevant. The error message seems to suggest line 1 and col 1, but it’s only a {.

Turns out, the fix is (note the curly braces).

1
2


- docker ? import ./nix/docker.nix,
+ docker ? import ./nix/docker.nix {},

It was not intuitive for me to figure that out.

Confusing Commands

There’s quite a few Nix commands:

nix
nix-build
nix-channel
nix-collect-garbage
nix-copy-closure
nix-daemon
nix-env
nix-hash
nix-info
nix-instantiate
nix-prefetch-url
nix-shell
nix-store
nixos-build-vms
nixos-container
nixos-enter
nixos-firewall-tool
nixos-generate-config
nixos-help
nixos-install
nixos-option
nixos-rebuild
nixos-version

Now you can probably guess what most of these do, but that’s not what I mean. I want to upgrade my Nix daemon. How do I do it? nix upgrade? Nope.

Nix, the package manager

What I would have expected for versioning

As I understand it, when you pick a package from nixpkgs (e.g. pkgs.nginx), you’re picking what ever happens to be defined as the version in the HEAD commit in NixOS/nixpkgs repo. Using a Nix Flake can freeze the commit you’re looking at, but that is all of nixpkgs. You’re still at the mercy of whatever the nixpkg maintainer used as a version.

That version could be out of date or it could even be an unofficial release candidate (e.g. this commit adopted an rc version for the onedrive package). Or maybe the docker uses 27.x which is not officially supported by your Kubernetes engine RKE1.

What are my options? With some packages like Docker, Nixpkgs continues to maintain the old versions, so I can just change from pkgs.docker to pkgs.docker_26, but not every package has this luxury. Nor is it clear how I actually use it when I use:

1

virtualisation.docker.enable = true;

In the onedrive example, the change was unexpected. It appears that it’s only a release candidate in the unstable Nix channel, but unstable seems to be the default.

I could always just copy the derivation into my own repository and never deal with any issues, but that’s non-trivial.

I see the nixpkgs style of versioning to be fundamentally not user-friendly. I think I’d prefer something like programming language dependency modeling (think Ruby Bundler’s Gemfile or Python’s Pipfile) where I can do:

1
2


containerd =~ 24.0.0
firefox = *

Right now, the upgrade story brings risk for unexpected changes. I do a nix flake up and nixos-rebuild switch and things just poof change.

How do I even override something?

I’m trying to use the Kubernetes NixPkg, but I needed to modify it to:

Pin the control plane and worker versions so I can bump the control plane before bumping the worker nodes
Pin the pause image so it doesn’t get accidentally deleted breaking my cluster
Not delete the CNI binaries automatically every time Kubelet starts

How do I pin NixPkgs?

To pin the versions, I found posts that said I could just import a specific commit of NixPkgs like this and then update the commit as I desire:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


diff --git a/flake.nix b/flake.nix
index a4f5dc0..2ff8429 100644
--- a/flake.nix
+++ b/flake.nix
@@ -3,6 +3,8 @@
 
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs";
+    # Grab a specific version of the NixPKGs so we don't accidentally update K8s
+    nixpkgs-k8s.url = "github:NixOS/nixpkgs/78324291425a318af7b6fe08ce0646291f5588db";
    nixos-hardware.url = "github:NixOS/nixos-hardware/master";
    disko = {
      url = "github:nix-community/disko";
@@ -10,7 +12,7 @@
    };
  };
 
-  outputs = { self, nixpkgs, disko, ... }@inputs: {
+  outputs = { self, nixpkgs, nixpkgs-k8s, disko, ... }@inputs: {
    nixosConfigurations.srv5 = nixpkgs.lib.nixosSystem {
      specialArgs = {inherit inputs;};
      modules = [
diff --git a/parts/kubernetes.nix b/parts/kubernetes.nix
index 5947840..47207ed 100644
--- a/parts/kubernetes.nix
+++ b/parts/kubernetes.nix
@@ -1,4 +1,4 @@
-{ config, lib, pkgs, ... }:
+{ config, lib, pkgs, nixpkgs-k8s, ... }:

It’s not exactly what I want which is to pin to a specific Kubernetes Major.Minor versions, but it’s fine. However, it still doesn’t make any sense:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


{ config, lib, pkgs, nixpkgs-k8s, ... }:
let
 # ...
in
{
 services.kubernetes = {
 kubelet = {
 # ... How does one specify that kubelet is from nixpkgs-k8s, not the default pkgs?
 };
 };
}

What I think is happening is that the nixpkgs-k8s parameter does point to the expected commit, but nixpkgs is composed of both configuration and packages. The configuration services.kubernetes.kubelet works off the latest commit.

My first assumption, until I was actually writing this post, was that I’d have to duplicate the services/cluster/kubernetes files and here where it builds the systemd service, I somehow modify the path to kubelet from ${top.package}/bin/kubelet to ${nixpkgs-k8s.services.kubernetes.package}

However, I now see a services.kubernetes.package option that I think I can override. But how? Maybe nixpkgs-k8s.services.kubernetes.package?

1
2
3
4
5
6
7


{ config, lib, pkgs, nixpkgs-k8s, ... }:
let
 # ...
in
{
 services.kubernetes = {
 package = nixpkgs-k8s.services.kubernetes.package;

Nope, I get:

1

error: attribute 'nixpkgs-k8s' missing

I’m not really sure what to do to fix this. Maybe I need to vendor the entire kubernetes/ folder myself, but I look at it and I will have to make a number of modifications to get it to work to change variable references.

Conclusion

Nix is a fascinating idea. The ability to declaratively define my entire OS is great for servers. I can have all my servers look exactly the same and easily create ones as I want. I like the idea of being able to Git revision control changes. It really satisfies my itch to centralize and cleanly define configuration.

However, I find working with the Nix language and Nix the package manager to be extremely frustrating. Usually there’s a learning curve that I can get over and become proficient enough at it, but Nix I struggle with. The Nix language is just weird enough to be hard to work with, and I don’t really have faith in Nix, the package manager, not to just blow something up that I critically need.

Proxy ARP is broken on Unifi U7 Lite

Sat, 04 Oct 2025 14:28:00 -0800

For several years, I had 2x Unifi U6 Lite Access Points and they worked great. I had a special Wi-Fi network for my phones and laptops with a number of settings enabled but then I upgrade to the U7 Lite and immediately started having issues where my phone would disconnect. I got frustrated enough to break out my handy tool box to figure what was going wrong.

As it turns out, proxy ARP was breaking DNS traffic. It was able to send DNS queries to my DNS server, but responses were getting stuck because it wasn’t able to send the response back.

ARP is necessary to translate an IP address into the Ethernet MAC Address that switches actually use to forward packet when on the same LAN.

First Attempt using airomon-ng

I first attempted to packet capture the wireless packets between my phone and access point to see what is actually causing the phone to think it’s disconnected. Using my linux laptop, I used airmon-ng to listen on my network channel.

1
2
3


airmon-ng check kill
airmon-ng start wlp1s0
airodump-ng wlp1s0mon --channel 155 --band a -w output.pcap

I had some difficulty figuring out what channel to use for this given it was an 80Mhz channel width and airodump only seemed to support up to a 40Mhz channel width. This command was able to capture some packets.

Decrypting WPA3

Now that I have a Wireshark packet capture, I tried to decrypt it in Wireshark. Wireshark supports decryption if you have the encryption key, but WPA3 makes this more difficult because each device has it’s own encryption key so you need the “Pairwise Master Key (PMK)” from either the AP or the device. It’s not easy to get it out of Android, but luckily my AP has SSH access.

I SSH’d into the AP and ran:

1
2
3


ps | grep hostapd

kill {pid} && hostapd -S -g /var/run/hostapd/global -P /var/run/host -d K

I reconnected my phone to Wi-Fi. At some point the logs contain this:

1

WPA: PMK - hexdump(len=32): bb 6e 4a 93 7b 2e e2 29 8b d3 29 24 02 6e b0 0e f4 d5 99 d6 f6 d2 06 60 a1 e1 5f 53 01 2c 37 17

To load this into Wireshark, go to Edit > Preferences > Protocols > IEEE 802.11 > Decryption Keys Edit. Add a WPA-PSK:

1

bb6e4a937b2ee2298bd32924026eb00ef4d599d6f6d20660a1e15f53012c3717

In theory, Wireshark should now be able to decrypt the Wi-Fi packets. However, what I noticed is that airodump-ng did not seem to dump all the data packets, possible due to a limitation in my Intel Wi-Fi card.

A second look at the HostAPd logs

I went back to log at the logs on the access point and just looked at the logs being generated:

1

tail -f /var/log/messages

I saw something interesting every time the phone would report the Wi-Fi doesn’t work. It was sending DNS queries to my pi-hole server and the AP was detecting that there was no response back.

1
2
3
4
5
6


kern.warn kernel: [STA_TRACKER] DNS request timed out; [STA: 12:f3:85:dd:92:95][QUERY: play.googleapis.com.] [DNS_SERVER :192.168.6.2] [TXN_ID 81e7] [SRCPORT 8571]
kern.warn kernel: [STA_TRACKER] DNS request timed out; [STA: 12:f3:85:dd:92:95][QUERY: play.googleapis.com.] [DNS_SERVER :192.168.6.2] [TXN_ID 81e7] [SRCPORT 8571]
kern.warn kernel: [STA_TRACKER] DNS request timed out; [STA: 12:f3:85:dd:92:95][QUERY: connectivitycheck.gstatic.com.] [DNS_SERVER :192.168.6.2] [TXN_ID 65e5] [SRCPORT 63180]
kern.warn kernel: [STA_TRACKER] DNS request timed out; [STA: 12:f3:85:dd:92:95][QUERY: connectivitycheck.gstatic.com.] [DNS_SERVER :192.168.6.2] [TXN_ID 65e5] [SRCPORT 63180]
kern.warn kernel: [STA_TRACKER] DNS request timed out; [STA: 12:f3:85:dd:92:95][QUERY: connectivitycheck.gstatic.com.] [DNS_SERVER :192.168.6.2] [TXN_ID b304] [SRCPORT 55085]
daemon.info stahtd: stahtd[28828]: [STA-TRACKER].stahtd_dump_event(): {"message_type":"STA_ASSOC_TRACKER","mac":"ab:cd:ef:ab:cd:ef","vap":"ath4","event_type":"dns timeout","assoc_status":"0","query_0":"play.googleapis.com.","query_server_0":"192.168.6.2","query_1":"www.google.com.","query_server_1":"192.168.6.2","query_2":"play.googleapis.com.","query_server_2":"192.168.6.2","query_3":"www.google.com.","query_server_3":"192.168.6.2"}

Is it DNS?

DNS is looking awfully suspicious right now and it is frequently blamed for network issues. However, if I check Pi-Hole (running at 192.168.6.2 mentioned above), it shows that it successfully resolved queries in the query log and sent a response.

tcpdump everything

Time to start packet capturing everything. My network looks like this:

DNS queries are sent from my phone through the AP through the router to the Pi-hole. My pi-hole is being via out of Kubernetes, but the responses go directly to the sender because my server is on two networks. It’s complicated, but it does not break any Ethernet standards.

I ran tcpdump on the Wi-Fi access point, on the router, and on the server. All three points show the DNS queries going to the server, but no responses back. The access point shows this cpature:

1
2
3
4
5
6
7
8


13:54:11.331369 ath4 P IP 192.168.2.62.32550 > 192.168.6.2.53: 12773+ AAAA? example.com. (35)
13:54:11.331918 eth0 B ARP, Request who-has 192.168.2.62 tell 192.168.2.3, length 46
13:54:11.331953 ath4 Out ARP, Request who-has 192.168.2.62 tell 192.168.2.3, length 46
13:54:11.331953 ath4 P IP 192.168.2.62.1558 > 192.168.6.2.53: 41230+ A? example.com. (35)
13:54:11.331986 ath2 Out ARP, Request who-has 192.168.2.62 tell 192.168.2.3, length 46
13:54:11.331993 eth0 Out IP 192.168.2.62.1558 > 192.168.6.2.53: 41230+ A? example.com. (35)
13:54:11.332003 ath0 Out ARP, Request who-has 192.168.2.62 tell 192.168.2.3, length 46
13:54:11.332020 br0 B ARP, Request who-has 192.168.2.62 tell 192.168.2.3, length 46

The server is trying to figure out how to send the response back to the client by performing an ARP Request to say where is this IP address? The switch knows where the MAC address is and forwards the packet to the AP. The AP seems like it sends it to the phone, but the phone either doesn’t respond or doesn’t receive it even though the packet capture shows it going out the ath4 interface which is the Wi-Fi adapter.

Proxy ARP

That gave me the idea to check the Wi-Fi network settings. Unifi exposes a number of different options and I already turned off the more problematic settings to see, but didn’t turn off Proxy ARP.

Unifi says Proxy ARP Reduces airtime usage by allowing APs to “proxy common broadcast frames as unicast. This can improve latency, but may cause connectivity issues in some networks.

Clearly, the “may cause connectivity issues” is right. This is unusual because I had the setting on for all my U6 Lites without any issue, but as soon as I get a U7 Lite, it breaks.

It’s now been over a week after disabling this option without any issues with my phone now. For reference, I’m running firmware v8.0.49 in case they fix it later.

Auto enable user namespaces in Kubernetes

Mon, 29 Sep 2025 20:00:00 +0000

When you run a container, the process IDs are namespaced and different in the container vs the host, the network stack is namespaced, the file system mounts are namespaced, but a process running as root in the container is running as root outside the container. This is risk because many privilege escalation vulnerabilities in Linux can be exploited because of this common user id.

Linux user namespaces aims to mitigate risks with running a process as root or other shared user ID where a vulnerability could allow a containerized process to escape a namespace and have privileges in the host. For example, process running as root in a container, would be marked as root outside the host without user namespaces or a process that has a UID that exists on the host.

User namespaces attempt to fix this by saying UID=0 inside the container actually is UID=12356231 on the host. Thus, a breakout is not as bad as it could be without user namespaces.

In this post, I’m going to walk through how I use Kyverno, a Kubernetes native policy system to easily enable user namespaces in pods where it can be.

Release Schedule

In release v1.30, Kubernetes announced beta support for User namespaces. In release v1.32, Kubernetes enabled it by default.

If you’re running v1.30-v1.31, then enable the feature gate on the APIServer with the --feature-gates=UserNamespacesSupport=true arg.

Thinking about policies

Kyverno policies are defined as Kubernetes resources, then when any workload resources are created or changed, Kyverno receives a webhook call and makes changes based on the policy.

What I want is to automatically set PodSpec.hostUsers=false when a pod is created. However, due to limitations, I can’t use it with hostNetwork: true, hostIPC: true, hostPID: true, or volumeDevices. Also, any volumes that get mounted into the pod have to support idmap. If I set hostUsers: false on every pod, then I’ll get a bunch of failures and have a bad day.

I’m going to walk through how I created the policy. If you’re just interested in the final policy, Skip below.

Creating the policy

Mutate all the things

First, I start with a basic policy that enables users namespaces on all pods that are created in the cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 annotations:
 pod-policies.kyverno.io/autogen-controllers: none
 name: enable-userns-when-able
spec:
 background: false
 rules:
 match:
 any:
 - resources:
 kinds:
 - Pod
 mutate:
 patchStrategicMerge:
 spec:
 hostUsers: false
 name: enable-userns
 preconditions:
 all:
 - key: '{{ request.operation || ''BACKGROUND'' }}'
 operator: Equals
 value: CREATE
 validationFailureAction: Audit

However, this will cause some pods to fail to create because user namespaces isn’t supported on all types of volumes and hostNetwork.

Skip for pods with other host namespacing

User namespacing isn’t supported with pods using the host network or host PIDs. My guess is because those would enable a container to see user ids that it can’t know about.

I want to exclude pods that have these values set. This can be done with a precondition. Kyverno’s precondition language gets to be quite strange with complex preconditions, but these are trivial.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


preconditions:
 all:
 # ... previous precondition
 - key: '{{ request.object.spec.hostUsers || ''false'' }}'
 operator: NotEquals
 value: 'true'
 - key: '{{ request.object.spec.hostIPC || ''false'' }}'
 operator: Equals
 value: 'false'
 - key: '{{ request.object.spec.hostPID || ''false'' }}'
 operator: Equals
 value: 'false'
 - key: '{{ request.object.spec.hostNetwork || ''false'' }}'
 operator: Equals
 value: 'false'

Now when I create any pods with hostNetwork=true or hostIPC or explicitly setting hostUsers=true, then nothing is changed.

Problems with PVCs

However, I still have problems when I start working with volumes. If I mount a hostPath volume, it works because your host volumes support user namespaces, but not all volumes do support them. For example, Longhorn, my storage provider in my home lab, uses NFS (Network File System) to mount volumes and it didn’t support idmap:

1
2
3


failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: err
or during container init: failed to fulfil mount request: failed to set MOUNT_ATTR_IDMAP on /var/lib/kubelet/pods/7ea01f58-57f9-4504-87b5-898ca80bd980/volumes/kube
rnetes.io~csi/pvc-48bee7b4-82b5-4119-a927-ceb0b0939c20/mount: invalid argument (maybe the filesystem used doesn't support idmap mounts on this kernel?)

Looking further, I believe this issue is limited to just ReadWriteMany volumes (volumes that can be mounted on multiple nodes at once) as other volumes are able to work with user namespaces.

I also encountered issues with pods that mount devices from the host’s /dev, like my Home Assistant that talked to USB Zigbee adapters.

I need a way to skip mutating pods that use volumes that don’t support user namespaces.

Skip for unsupported volumes

This is where Kyverno’s policy language gets very messy and unintuitive.

Skip host hardware devices

First, we check to see if there’s any mounts that use hostPath to mount something from under the host’s /dev/ folder. If it finds one or more, it skips mutating because user namespaces isn’t supported. This uses a language called JMESPath to search the JSON/YAML file kind of like XPath.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


spec:
 rules:
 - context:
 - name: hasdevice
 variable:
 default: 0
 jmesPath: >-
 request.object.spec.volumes[?hostPath].hostPath.path[?starts_with(@,
 '/dev/')] | length(@)
 # mutate:
 # match: 
 preconditions:
 # - ... other preconditions
 - key: '{{ hasdevice }}'
 operator: Equals
 value: 0

Skip for Longhorn

Next, we look for any volume mounts that reference a Longhorn RWX volume. A pod doesn’t directly say what kind of PVC is being mounted. It just says the name of the PVC. What can we do to figure this out?

1
2
3
4
5
6
7


apiVersion: v1
kind: Pod
spec:
 volumes:
 - name: data
 persistentVolumeClaim:
 claimName: paperless

Luckily, Kyverno has a feature that calls back to the Kubernete’s API to fetch one or more resources. I have the name of the PVC, can this tell me what kind of volume I have?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# kubectl get pvc -n paperless paperless -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 annotations:
 pv.kubernetes.io/bind-completed: "yes"
 pv.kubernetes.io/bound-by-controller: "yes"
 volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
 volume.kubernetes.io/storage-provisioner: driver.longhorn.io
 labels:
 recurring-job-group.longhorn.io/default: enabled
 recurring-job.longhorn.io/weekly: enabled
 name: paperless
 namespace: paperless
spec:
 accessModes:
 - ReadWriteOnce
 - ReadWriteMany
 storageClassName: longhorn

It has accessModes, so we can use that to skip mutating RWX volumes. The spec.storageClassName looks relevant at first, but you can create more Storage Classes with any name that uses Longhorn and it’s common to define classes with different node or disk selectors, so while it would work, it’s fragile and easy to break.

There’s also the the metadata.annotations."volume.kubernetes.io/storage-provisioner", but this isn’t present on my older volumes–only the beta annotation is there and looking at deprecated annotations feels dirty. Thus, I’d have to look at both annotations.

The following will tell Kyverno to go back to Kubernetes and fetch all PVCs in the same namespace, then locally filter using JMESPath to find all PVCs that are managed by Longhorn and use ReadWriteMany and stores the names in a variable called longhornpvcs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


spec:
 rules:
 - context:
 - apiCall:
 jmesPath: >-
 items[?((metadata.annotations."volume.kubernetes.io/storage-provisioner" == 'driver.longhorn.io' || metadata.annotations."volume.beta.kubernetes.io/storage-provisioner" == 'driver.longhorn.io') && contains(spec.accessModes, 'ReadWriteMany'))].metadata.name
 method: GET
 urlPath: /api/v1/namespaces/{{ request.namespace }}/persistentvolumeclaims
 name: longhornpvcs
 # match:
 # mutate:

Then, in the preconditions, we iterate over all the volumes in the Pod, gets the names and sees if any of them exist in the set I defined above. If any of them are, it skips mutation. Additionally, skips mutation for any volumes from any provisioner that mounts the volume as a volumeDevice, which is a low level mount and not supported by user namespaces.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


spec:
 rules:
 - # context:
 # match:
 # mutate:
 preconditions:
 all:
 # ... other preconditions
 - key: >-
 {{
 request.object.spec.volumes[?persistentVolumeClaim].persistentVolumeClaim.claimName
 || `[]` }}
 message: Longhorn PVCs may not support user namespaces
 operator: AllNotIn
 value: '{{ longhornpvcs }}'
 - key: '{{ request.object.spec.containers[?volumeDevices] | length(@) }}'
 message: "volumeDevices don't support support user namespaces"
 operator: Equals
 value: 0

The Final Policy

Here’s the final policy put together:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 annotations:
 kyverno.io/kyverno-version: 1.9.0
 pod-policies.kyverno.io/autogen-controllers: none
 policies.kyverno.io/category: Pod Security
 policies.kyverno.io/description: This policy ensures that user namespaces are enabled for pods that can
 policies.kyverno.io/severity: medium
 policies.kyverno.io/subject: Pod
 policies.kyverno.io/title: Enable User Namespaces when possible
 name: enable-userns-when-able
spec:
 background: false
 rules:
 - context:
 - name: hasdevice
 variable:
 default: 0
 jmesPath: >-
 request.object.spec.volumes[?hostPath].hostPath.path[?starts_with(@,
 '/dev/')] | length(@)
 - apiCall:
 jmesPath: >-
 items[?((metadata.annotations."volume.kubernetes.io/storage-provisioner" == 'driver.longhorn.io' || metadata.annotations."volume.beta.kubernetes.io/storage-provisioner" == 'driver.longhorn.io') && contains(spec.accessModes, 'ReadWriteMany'))].metadata.name
 method: GET
 urlPath: /api/v1/namespaces/{{ request.namespace }}/persistentvolumeclaims
 name: longhornpvcs
 exclude:
 any:
 - resources:
 namespaces:
 - kube-system
 - longhorn-system
 - calico-system
 match:
 any:
 - resources:
 kinds:
 - Pod
 mutate:
 patchStrategicMerge:
 spec:
 hostUsers: false
 name: enable-userns
 preconditions:
 all:
 - key: '{{ request.operation || ''BACKGROUND'' }}'
 operator: Equals
 value: CREATE
 - key: '{{ request.object.spec.hostUsers || ''false'' }}'
 message: Skipping because hostUsers is explicitly set to true
 operator: NotEquals
 value: true
 - key: '{{ request.object.spec.hostIPC || ''false'' }}'
 operator: Equals
 value: 'false'
 - key: '{{ request.object.spec.hostPID || ''false'' }}'
 operator: Equals
 value: 'false'
 - key: '{{ request.object.spec.hostNetwork || ''false'' }}'
 message: "User namespaces can't be used when hostNetwork=true"
 operator: Equals
 value: 'false'
 - key: '{{ hasdevice }}'
 operator: Equals
 value: 0
 - key: '{{ request.object.spec.containers[?volumeDevices] | length(@) }}'
 message: "volumeDevices don't support support user namespaces"
 operator: Equals
 value: 0
 - key: >-
 {{
 request.object.spec.volumes[?persistentVolumeClaim].persistentVolumeClaim.claimName
 || `[]` }}
 message: Longhorn PVCs may not support user namespaces
 operator: AllNotIn
 value: '{{ longhornpvcs }}'
 validationFailureAction: Audit

See it in action

The policy only applies to new pod creation, so existing pods are not modified. To see it, we need to redeploy pods.

To see which pods are using user namespaces with kubectl, run the following command. If the HostUsers column shows <none> or true, then it’s not. If it shows false, then the pod has it’s own separate user namespace.

1

kubectl get --all-namespaces pods -o custom-columns=Namespace:.metadata.namespace,Name:.metadata.name,HostUsers:.spec.hostUsers

Output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


NAMESPACE               NAME                                                           USERNS
authelia                authelia-6dd4b4b75c-zvm7m                                      false
calico-apiserver        calico-apiserver-7dbc7798c5-nvrqp                              <none>
calico-system           calico-kube-controllers-5778576475-b6gr8                       <none>
calico-system           calico-node-92t8f                                              <none>
calico-system           calico-typha-55dfc8c999-ckldk                                  <none>
calico-system           csi-node-driver-9jhvb                                          <none>
cattle-system           cattle-cluster-agent-746f6d788f-h6n2q                          <none>
cattle-system           kube-api-auth-cqrcz                                            <none>
cattle-system           rancher-webhook-5d4b7cd966-phdj2                               <none>
cattle-system           system-upgrade-controller-57b66d6cbd-6p92n                     <none>

Guaranteed Quality of Service in my Home Lab

Wed, 17 Sep 2025 15:35:00 +0000

A few times in my Kubernetes clusters, I’ve encountered situations where some process consumes all the CPU or RAM which starves other services for critical services. For example, in one situation, Longhorn consumed all CPU and RAM and my pi-hole running on the same machine stopped being able to process DNS requests. Other issues have included having to shut down one of my worker nodes and the other nodes not having enough capacity to take on pods and important pods not getting scheduled or even a mistake when I changed the pod selector labels and Kubernetes just spawned thousands of pods.

The graph below shows Disk I/O of a node with excessive disk writes because the OS is swapping RAM out to desk and back.

My home lab servers are now running what I consider to be “business critical” services and I don’t want those to be impacted. Kubernetes has several different knobs we can use to improve this such as leveraging Linux’s cgroups to ensure that specific pods get a certain amount of CPU and RAM. It also supports prioritization, so that certain pods get scheduled and others get evicted if there isn’t enough space.

Or even lately, I’ve been hitting the max pod limit of 110 pods on my single-node cluster. Not everything is important and I want to make sure certain cron jobs always run even if I’m running some low-priority jobs. Turns out it is possible to be running 110 different pods.

Image Below: A sad worker node valiantly trying to keep operating without enough RAM. The breaks in metrics is when Prometheus is unable to scrape metrics because it didn’t get a chance to run.

In this post, I’m going to walk through the steps that I took to improve reliability by using Kubernetes’ resource management and scheduling features. My clusters run on a limited number of hosts (only one node for my home and three nodes for my public cluster) and do not automatically scale up or down nodes, so I need to ensure that everything fits onto the clusters without starving any resource.

The problem

Each of the incidents that I alluded to earlier were caused by me trying to schedule too many services for the given nodes. Exceeding the total RAM space causes the kernel to start swapping memory out to disk which slows down services as memory pages have to keep getting swapped in and out and additionally prevents the kernel from caching files in RAM.

In this chart, the OS has run out of RAM and Prometheus is unable to get time on the CPU to scrape metrics. Before Prometheus can run any code, it has to wait for the Kernel to swap in pages that it needed. This was bad because it was starving Pi-hole’s DNS server for CPU time and prevented Internet access.

Overscheduling the CPU with too much work means that there’s threads awaiting time on the CPU, but not enough cores. Like RAM swapping, this will increase latency of operations.

There are other ways to saturate a host like disk or network I/O, but I’ll focus on RAM and CPU because Kubernetes provides resource management for these two limits.

Resource requests and limits

The first thing is setting resource requests and optionally limits on workloads. A resource request states that a given container should only be allocated on a node if there’s enough un-reserved space. So a container that requests 4GB should not be allocated on a node with 8GB of RAM that already has 6GB of RAM dedicated to other workloads.

A limit states the maximum amount of a resource a given container should be allocated. Memory limits are given to the container so it knows how much it can allocate, though exceeding it will result in the container being killed. Software can be a little bit unpredictable in how it uses resources, it might spike up and settle back down faster than Prometheus can collect metrics, so be conservative in your limits.

If you don’t set a limit, then the container can use as much as it wants.

Can not setting a limit cause problems for other services?

Resource requests and limits use Linux cgroups under the hood which tells the Linux scheduler how much time and memory should be provided to a process or process group (a process group is a hierarchy of processes.) If you don’t use resource requests at all, then one process can use memory and push another process out of memory (unless it gets OOMKilled by the kernel.)

A memory request sets the memory.min cgroup setting which tells the Kernel to avoid reclaiming memory from this container until there’s no other process to recover RAM from. This means that the Kernel will generally scavenge RAM from other places before impacting the processes you really care about.

By default, Kubernetes also sets the memory.swap.max cgroup setting to 0 which prevents the Kernel from swapping memory pages out for the given container, however as in the example above, there’s still plenty of other pages that are necessary that aren’t directly in the container’s space, there’s a lot of system daemons, file caching, and other processes that are crucial for operation that could start swapping. However, this can be mitigated by reserving space for those processes directly.

So, not setting a limit can still cause issues, however setting requests gives you a lot of protection against over-scheduling until things get really bad.

Picking numbers

Memory

To pick my resource requests, I go to Prometheus after a service has been running for a bit of time and look at it’s RAM and CPU usage. For example, looking at the following graph, I’d say homeassistant should have 1GB for a request and the limit.

When picking these numbers, consider the workload. Does it have spiky operations where it pulls in a bunch of data, then clears it out later? Make sure to accommodate that usage. Does memory trend up? Maybe you’ve got a memory leak or maybe it’s just caching more. Many processes will adapt to memory limits and perform garbage collections more frequently (at the cost of more CPU usage) or just reduce it’s caching to adapt.

Prometheus Query: container_memory_usage_bytes{container!="POD",container!=""}

CPU

Let’s try to do the same thing with CPU. Starting with the below graph, I can see that homeassistant uses about 7ms of CPU per second and seems to peak at 68ms per second. It’s per second because irate averages the increase over the interval to get the per second rate. Prometheus Query: irate(container_cpu_usage_seconds_total{container!=""}[1m])

Setting a CPU request is safe because it defines the minimum amount of CPU time reserved for the process. It’s measured in number of CPUs, so a request of 1 reserves an entire core for the container, a request of 2 gives two cores, and a request of 256m gives 1/4 of a core. For this example, I went with resource.requests=256m so it shares a CPU core with other process.

However, there’s peril with setting a limit. As a turns out CPU utilization is misleading and tons of posts that say you should or shouldn’t use CPU limits.

CPU is trickier than memory because CPU usage in Prometheus is measured in milliseconds, but then averaged over seconds or minutes in graphs. So instead, if we change from irate to increase which does not average it and shows the pure increase from one minute to the next, it shows that we used an average of 540ms of CPU time per minute.

If that were to execute all in a 1 second block, then setting a limit could mean the process is stuck waiting for CPU time when there is idle time available. If you’re not sure how the process behaves, it’s probably safer not to set a limit.

The biggest benefit with setting a limit is if you have the limits equal to requests, then Kubernetes will mark the pod with the guaranteed QoS meaning that it’s always the last pods to get evicted by Kubernetes or to face the wrath of the kernel.

Defining them in YAML

Resource requests and limits are easy to set in YAML:

1
2
3
4
5
6
7
8
9


spec:
 containers:
 - name: homeassistant
 resources:
 limits:
 memory: 1Gi
 requests:
 cpu: 256m
 memory: 1Gi

Defining resource requests and limits gives the Kubernetes scheduler information on how heavy workloads are and ensures that there’s space to schedule them, but at this point it’s just first-come-first-serve. The first pod scheduled wins.

Kubectl command to review current settings

This command is useful for viewing all pods and their assigned requests:

1

kubectl get --sort-by=.spec.priorityClassName --all-namespaces pods -o=custom-columns=STATUS:.status.phase,NAME:.metadata.name,NAMESPACE:.metadata.namespace,PRIORITY-CLASS:.spec.priorityClassName,REQUEST-MEM:.spec.containers[*].resources.requests.memory,REQUEST-CPU:.spec.containers[*].resources.requests.cpu --field-selector status.phase=Running

Priority Classes

Priority classes then give the Kubernetes scheduler information to know which pods have to be running and which ones aren’t. It depends on the information provided by resource requests and limits to know when it has to prioritize. Official documentation can be found here. Prioritization ensures that my high-priority services, such as DNS, always gets scheduled and my lower-priority services, like my Blender distributed rendering service (Blendfarm) don’t kill my DNS.

Priority classes rank the services into most important to least important. Normally, I can run every single workload I care about, but defining these priority classes ensure that when I do some development work and spin up something to test, my scheduled jobs still run.

Identifying services

The first step is to identify all the services that are critical for functioning. I know that I need pi-hole for my Internet DNS traffic, Home Assistant and my smart home services, and Postgres for data storage.

But those services have dependencies themselves too. For example, pi-hole won’t start without:

Calico and Multus to setup networking and expose the service via BGP to my network
Longhorn to mount the data and config volumes
CoreDNS to handle DNS resolution for queries to my internal services

Other soft dependencies include:

ingress-nginx to be able to access the pi-hole UI
Vouch-proxy to handle access control to the UI

If I capture those dependency in a dependency graph, I end up a diagram like this just for pi-hole.

I start with pi-hole as critical for my network, then go through each dependency and ensure that it’s the same priority or higher priority.

My List

Starting from the lowest level, this is the list I came up with:

System
- containerd
- ssh
- k3s
Basic stuff
- Calico (networking CNI)
- CoreDNS
Storage
- Longhorn
- local-path-storage
Management (system-cluster-critical)
- cattle-cluster-agent - Needed so I can use the Rancher UI to manage my cluster
network-critical
- pi-hole
- metal-lb

Without everything above, I would just consider the entire home network non-functional and would quickly have frustrated family members and guests. That doesn’t mean I can’t run the next list. Things would still be broken, but at least I can get online to figure out how to fix the other issues.

smarthome-critical
- Home Assistant
- ZWaveJS
- MQTT
- Other Smart Home services
- Databases (Postgres, InfluxDB)
- Internal DNS Stuff (external-dns + etcd)
- ingress-nginx

My computer runs a lot of the automations in my house. Without them, I’d be sad, but still my light switches would work. Those services should still get priority over anything else.

smarthome-important
- Auth (Vouch Proxy)
- Voice Assistants for Smart Home (whisper, piper)

The smarthome-important category of services are still noticeable when they’re offline, but at least things work without them.

other-important - Things that I want to run instead of my random test services, but are not as important as things that actively get used by me on a daily basis
- Scheduled jobs

That led me to this diagram showing the dependencies of services and to think it’s not all of my workloads that I have running.

Creating the priority classes

Kubernetes comes with a few default classes and Longhorn also brings it’s own, so I have to create my own for the lower priorities:

1
2
3
4
5
6
7
8
9


kubectl get priorityclasses --sort-by='.value'
NAME VALUE GLOBAL-DEFAULT AGE PREEMPTIONPOLICY
lab-important 600000000 false 2d PreemptLowerPriority
smarthome-important 700000000 false 2y257d PreemptLowerPriority
smarthome-critical 800000000 false 2y257d PreemptLowerPriority
network-critical 900000000 false 2y257d PreemptLowerPriority
longhorn-critical 1000000000 false 497d PreemptLowerPriority
system-cluster-critical 2000000000 false 3y154d PreemptLowerPriority
system-node-critical 2000001000 false 3y154d PreemptLowerPriority

Creating a new one is easy and looks like this. Pick a value that’s smaller than the higher priority. Kubernetes convention seems to use multiples of 100_000_000 to give space to put your own priorities in.

1
2
3
4
5
6
7


apiVersion: scheduling.k8s.io/v1
description: Software critical for the minimum functioning of the home network. (e.g. DNS)
kind: PriorityClass
metadata:
 name: network-critical
preemptionPolicy: PreemptLowerPriority
value: 900000000

Making changes

Now that I’ve identified categories of services based on how important they, it’s time to start tagging and updating workloads and processes. I’m starting with the OS and working my way up.

The Operating System

Kubernetes doesn’t know how much space to reserve for the operating system itself. Think the kernel, sshd, systemd, and if you’re like me and use my server for a TV, a desktop environment. The kubelet provides a parameter --system-reserved to tell Kubernetes to ignore some of the resources when scheduling nodes.

I’ve got a 16 core CPU, so I’ll reserve two cores for the host OS and 4GB of RAM. I’m using K3s managed by Rancher, so the setting is controlled by the Cluster resource:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
 name: adamsnet
 namespace: fleet-default
spec:
 rkeConfig:
 machineSelectorConfig:
 - config:
 kubelet-arg:
 - kube-reserved=cpu=2,memory=4Gi

Rancher Management

Rancher System Agent

The Rancher system agent is used when you deploy a RKE2 or K3s cluster using the Rancher UI, like I do. It needs some CPU time reserved so that any changes to the cluster itself, like version upgrades, etc.. Management daemons should always get some reserved resources to ensure that they can make corrective actions, if necessary. For the system agent, I use systemd’s CPUShares property to give it some time.

1

systemctl edit rancher-system-agent.service

1
2


[Service]
CPUShares=200

Rancher Cluster Agent

Next, the Rancher Cluster Agent is used to proxy Kubernetes calls from the Rancher UI to the downstream cluster. Without this, I’d be unable to browse and make corrective actions. Another management component, but this one runs as a Kubernetes Pod. By default, it’s marked as system-cluster-critical which is the second highest priority. No need to change the priority class, just give it some resources:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
 name: adamsnet
 namespace: fleet-default
spec:
 clusterAgentDeploymentCustomization:
 overrideResourceRequirements:
 requests:
 cpu: 256m
 memory: 1Gi

Calico

Now we’re into Kubernetes workloads that need to be updated using the priorityClassName field on the PodSpec.

Calico provides networking for all pods. Without it, pods wouldn’t start. This reserves some CPU and memory for them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


---
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
 name: default
spec:
 calicoNodeDaemonSet:
 spec:
 template:
 spec:
 containers:
 - name: calico-node
 resources:
 requests:
 cpu: 200m
 memory: 256Mi
 typhaDeployment:
 spec:
 template:
 spec:
 containers:
 - name: calico-typha
 resources:
 limits:
 memory: 128Mi
 requests:
 cpu: 100m
 memory: 64Mi
---
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
 name: default
spec:
 apiServerDeployment:
 spec:
 template:
 spec:
 containers:
 - name: calico-apiserver
 resources:
 requests:
 cpu: 30m
 memory: 64Mi
 priorityClassName: system-cluster-critical

Postgres

I deploy Postgres using CloudNativePG which uses a custom resource to provision my Postgres database. Resource requests and priorities can be set like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: postgres
spec:
 priorityClassName: cluster-important
 resources:
 requests:
 memory: 2Gi
 cpu: 512m

Other workloads

Anything else is updated directly on the PodSpec

1
2
3
4
5
6
7
8
9


apiVersion: apps/v1
kind: Deployment
metadata:
 name: homeassistant
 namespace: smarthome
spec:
 template:
 spec:
 priorityClassName: smarthome-critical

Now it’s just a matter of working through every single workload including Deployments, DaemonSets, CronJobs, and StatefulSets to mark them with resources and priorities. You can pick whichever order you want, either most important or by the biggest consumer.

Conclusion

There’s a lot that goes into resource management in a Kubernetes cluster when running many different services. Kubernetes depends on the values for priorities and resource requests you provide on the services to decide how to schedule and run pods.

Resource requests tell Kubernetes whether there’s space to fit a given workload on the cluster.
Resource limits define the maximum size of a workload and prevent it from exceeding that.
Priority classes are used when the resource requests exceed total node space

References

Better Vault for Postgres access in my Home Lab

Tue, 09 Sep 2025 22:39:00 +0000

In my previous post on Vault, I showed how Hashicorp’s Vault can be used to protect important passwords, static passwords that don’t change frequently. Vault can do much more than this and can even automatically create temporary accounts and rotate passwords for database users.

Today, I’m using long-lived passwords that I generate once when I add a new service, I, along with most people, just insert those passwords into the environment like this:

1
2
3
4
5
6


spec:
 containers:
 - env:
 - name: DATABASE_URL
 value: >-
 postgresql://username:mypassword@postgres:5432/database

That’s not secure at all. While you can store them in Kubernetes Secrets, they’re not encrypted by default. Kubernetes can encrypt secrets, but they’re open to anybody with access to the cluster. The passwords are easily accessible to anybody with access to Kubernetes and are never rotated. This simply won’t do. In this post, I’m going to walk through how I switch to Vault for

Postgres

Vault natively supports Postgres and can either maintain the password for a single Postgres role and rotate the password periodically (called static), or generate a new role for every single Pod that runs (dynamic.) With static, I have to go into DataGrip and create a role and setup the permissions manually, but dynamically should be able to create it all for me.

If you haven’t already created a Database backend, create one (Vault docs) using the UI or the API. Add a connection to your database. Make sure to use a privileged user with the ability to create users. I just use the postgres super user account.

I can’t figure out dynamic roles

Dynamic roles in postgres were confusing. I couldn’t figure out how to grant privileges in a given database. I was able to create my user with the following creation statements, but no matter what I tried it

1
2
3
4


create user "{{name}}" with encrypted password '{{password}}' valid until '{{expiration}}';
grant connect on database openwebui to "{{name}}";
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";
grant all on database openwebui to "{{name}}";

It would grant CONNECT to the db:

But this was insufficient to actually connect to the database and create/edit tables. Apparently Postgres configures the database at the connection level and does not allow me to grant this.

1
2


[2025-09-08 22:28:17] openwebui.public> select * from chat
[2025-09-08 22:28:17] [42501] ERROR: permission denied for table chat

Use a role

Looks like I can’t avoid pre-configuring the roles with Postgres. First, create a new role and grant it all appropriate privileges on the database:

1
2
3
4
5


-- On openwebui.public
create role "openwebui-role";
grant connect on database openwebui to "openwebui-role";
grant create, usage ON schema public to "openwebui-role";
grant all privileges on all tables in schema public TO "openwebui-role";

Bring it all together

Then create a Vault dynamic user like so:

Role Name: Whatever you want or namespace/serviceaccount. This naming convention will be important if you’re following my guide
Connection Name: Your PGSQL connection
Type of Role: dynamic
Creation Statements

1
2


create user "{{name}}" with encrypted password '{{password}}' in role "openwebui-role" valid until '{{expiration}}';
alter user "{{name}}" set role = "openwebui-role";

Now, if I click “Generate credentials” in Vault, I should get credentials that I login using DataGroup and view/edit table table.

Vault

Create the Vault role

So far, we’ve just created a secret that Vault can return, but our Kubernetes services can’t access these credentials, so we need to create a policy that allows a Vault user to access this Vault secret.

My previous approach of creating a new Kubernetes role and custom policy for every single Kubernetes service was become onerous. I want the ability to be able to add a new service to Vault without having to go through this work. If only services could automatically get access to their own secrets without being able to see other secrets. It’s possible with templated policies.

First, let’s create a common role in the Kubernetes authentication method that every pod can assume.

Authentication Method: Kubernetes
Name: k8srole
Alias name source: serviceaccount_name
Bound service account names: *
Bound service account namespaces: *

NOTE: Don’t assign any privileged policies because any service will be able to assume this role. If you want to be more careful, explicitly list the namespaces that run services that should have access in the role.

Create the Vault Policy

When I first started, I didn’t fully understand how templated policies in Vault worked. There were these parameters that I could use, like identity.entity.id, but it’s not clear how that looks in Kubernetes.

1
2
3


path "database/creds/???" {
 capabilities = ["read"]
}

To figure this out, I launched a pod with this Vault approle, and pulled the token out of /vault/secrets/token. Then used curl to call the API.

1

curl -H 'X-Vault-Token: hvs.CAE[...]' 'https://vault.example.com/v1/auth/token/lookup-self' | jq

That returned:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


{
 "request_id": "df26cb1d-8ca1-b53a-be17-1c96c12dd8c2",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": {
   "accessor": "jKE4BZeUR3N03qZMgsQJUtDA",
   "creation_time": 1729211189,
   "creation_ttl": 3600,
   "display_name": "kubernetes-default-my-service-account",
   "entity_id": "2a22a75d-be11-a77f-9ac4-8fe3cd16f4ee",
   "expire_time": "2024-10-18T01:26:29.249283537Z",
   "explicit_max_ttl": 0,
   "id": "hvs.CAE[...]",
   "issue_time": "2024-10-18T00:26:29.248036782Z",
   "last_renewal": "2024-10-18T00:26:29.249283637Z",
   "last_renewal_time": 1729211189,
   "meta": {
     "role": "test",
     "service_account_name": "my-service-account",
     "service_account_namespace": "default",
     "service_account_secret_name": "",
     "service_account_uid": "61c1afa0-dfa4-4c9f-8eec-7e5ebe0b6904"
   },
   "num_uses": 0,
   "orphan": true,
   "path": "auth/kubernetes/login",
   "period": 3600,
   "policies": [],
   "renewable": true,
   "ttl": 3503,
   "type": "service"
 },
 "wrap_info": null,
 "warnings": null,
 "auth": null
}

Then I fetched more details using the data.entity_id parameter:

1

curl -H 'X-Vault-Token: hvs.CAE[...]' https://vault.example.com/v1/identity/entity/id/2a22a75d-be11-a77f-9ac4-8fe3cd16f4ee | jq

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


{
 "request_id": "63e86ba7-39c3-66af-8048-a8c130fe20ab",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": {
   "aliases": [
     {
       "canonical_id": "2a22a75d-be11-a77f-9ac4-8fe3cd16f4ee",
       "creation_time": "2023-11-18T05:27:32.523910476Z",
       "custom_metadata": null,
       "id": "88853d50-7780-998b-fb78-c8141ded6a63",
       "last_update_time": "2023-11-18T05:27:32.523910476Z",
       "local": false,
       "merged_from_canonical_ids": null,
       "metadata": {
         "service_account_name": "my-service-account",
         "service_account_namespace": "default",
         "service_account_secret_name": "",
         "service_account_uid": "61c1afa0-dfa4-4c9f-8eec-7e5ebe0b6904"
       },
       "mount_accessor": "auth_kubernetes_398dc4e7",
       "mount_path": "auth/kubernetes/",
       "mount_type": "kubernetes",
       "name": "61c1afa0-dfa4-4c9f-8eec-7e5ebe0b6904"
     }
   ],
   "creation_time": "2023-11-18T05:27:32.523902982Z",
   "direct_group_ids": [],
   "disabled": false,
   "group_ids": [],
   "id": "2a22a75d-be11-a77f-9ac4-8fe3cd16f4ee",
   "inherited_group_ids": [],
   "last_update_time": "2023-11-18T05:27:32.523902982Z",
   "merged_entity_ids": null,
   "metadata": null,
   "name": "entity_0daf7612",
   "namespace_id": "root",
   "policies": []
 },
 "wrap_info": null,
 "warnings": null,
 "auth": null
}

Say I want to limit my policy to be able to access secrets named {{namespace}}|{{service_account_name}}, I use these parameters:

{[namespace}} = {{identity.entity.aliases.auth_kubernetes_398dc4e7.metadata.service_account_namespace}}. The auth_kubernetes_398dc4e7 is the name of the auth method specified in mount_accessor
{{service_account_name = {{identity.entity.aliases.auth_kubernetes_398dc4e7.metadata.service_account_name}}

Final Policy

Assuming I have a database storage end called database, I can grant access by creating a policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Dynamic users
path "database/creds/{{identity.entity.aliases.auth_kubernetes_398dc4e7.metadata.service_account_namespace}}|{{identity.entity.aliases.auth_kubernetes_398dc4e7.metadata.service_account_name}}" {
 capabilities = ["read"]
 allowed_parameters = {
 "password" = []
 }
}

# Static users
path "database/static-creds/{{identity.entity.aliases.auth_kubernetes_398dc4e7.metadata.service_account_namespace}}|{{identity.entity.aliases.auth_kubernetes_398dc4e7.metadata.service_account_name}}" {
 capabilities = ["read"]
 allowed_parameters = {
 "password" = []
 }
}

And if I want to grant read-only access to secrets under by-service/{{namespace}}/{{service_account}}/*

1
2
3


path "by-service/data/{{identity.entity.aliases.auth_kubernetes_398dc4e7.metadata.service_account_namespace}}/{{identity.entity.aliases.auth_kubernetes_398dc4e7.metadata.service_account_name}}/*" {
 capabilities = ["read", "list:]
}

NOTE: Your auth provider name will be different than auth_kubernetes_398dc4e7. Make sure you update it.

Create a policy with the above directives and name it something like k8s-default-policy. The go back to the Kubernetes role k8srole and add the policy under Generated Token’s Policies.

Using it in an app

How you get an app to use the Vault credentials will differ depending on the app itself.

Using environment variables

For applications that take the password using an environment variable and can’t use a file, I use Vault’s templating system to generate a file called /vault/secrets/env, then source it and invoke the process as normal.

For example, with OpenWebUI, I would do:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


apiVersion: apps/v1
kind: Deployment
spec:
 template:
 metadata:
 annotations:
 vault.hashicorp.com/agent-inject: 'true'
 vault.hashicorp.com/role: k8srole
 vault.hashicorp.com/agent-inject-template-env: >-
 {{- with secret "database/creds/openwebui" -}}
 export DATABASE_URL=postgres://{{ .Data.username }}:{{ .Data.password
 }}@postgres.datastore.svc.cluster.local.:5432/openwebui?sslmode=disable
 {{- end }}
 spec:
 containers:
 - command:
 - /bin/sh
 args:
 - '-c'
 - [ -f /vault/secrets/env ] && . /vault/secrets/env && /app/backend/start.sh

This generates a file that looks like:

1

export DATABASE_URL=postgres://v-kubernet-openwebu-[...]:[password]@postgres.datastore.svc.cluster.local.:5432/openwebui?sslmode=disable

Using files

Files are the easiest way. For example, Authelia can be configured using:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


spec:
 template:
 metadata:
 annotations:
 vault.hashicorp.com/agent-inject: 'true'
 vault.hashicorp.com/role: k8srole
 vault.hashicorp.com/agent-inject-template-db-password: >-
 {{- with secret "database/static-creds/authelia" -}}{{ .Data.password
 }}{{- end -}}
 spec:
 containers:
 - env:
 - name: AUTHELIA_STORAGE_POSTGRES_PASSWORD_FILE
 value: /vault/secrets/db-password

Conclusion

Thus far, I’ve create a reusable Vault role and Vault policy that that any Kubernetes pod with a service account can assume and login to Vault. It will only have access to secrets that match it’s name. Now it’s easy to onboard new services and grant secure access.

Git pushes can be surprising

Sat, 30 Aug 2025 23:28:00 +0000

I was recently working on an open source project (tryfi/hass-tryfi - A Home Assistant integration for pulling data from my dog’s collar using the TryFi API and I found out that Git pushes can behave in a surprising way after I accidentally pushed a bunch of testing commits to the wrong branch.

Background

In my workspace, I had two different remotes. One that tracked my own testing repo and one that was the official repo:

1
2
3
4
5


$ git remote -v
official    https://github.com/tryfi/hass-tryfi.git (fetch)
official    git@github.com:tryfi/hass-tryfi.git (push)
origin   https://github.com/ajacques/hass-tryfi.git (fetch)
origin   git@github.com:ajacques/hass-tryfi.git (push)

I had two branches: master (upstream to origin/master) and official-master (upstream to officialf/master). This enabled me to do testing work on my own repo and push it and when I thought it was ready, merge into the official repo. Yes these names are confusing. I’ll fix it later.

What was the surprise?

All hell broke loose when I was on my main-master branch and wanted to push some small commits to the tryfi/hass-tryfi repo:

1

git push official master

What I assumed would happen is that it would take my main-master branch commits and push them to main/master. That would make sense wouldn’t it? Turns out, that’s not what happened. It took all the juicy, barely tested commits on my master branch of the same name and pushed them to main/master.

Git Config

After cleaning up my mess, I then tried to figure out what I did wrong. Looking at the Git docs shows an interesting git config option: push.default that “Defines the action git push should take if no refspec is given (whether from the command-line, config, or elsewhere).” It defines the following options:

nothing - do not push anything (error out) unless a refspec is given. This is primarily meant for people who want to avoid mistakes by always being explicit.
current - push the current branch to update a branch with the same name on the receiving end. Works in both central and non-central workflows.
upstream - push the current branch back to the branch whose changes are usually integrated into the current branch (which is called @{upstream}). This mode only makes sense if you are pushing to the same repository you would normally pull from (i.e. central workflow).
tracking - This is a deprecated synonym for upstream.
simple - DEFAULT - push the current branch with the same name on the remote.
matching - push all branches having the same name on both ends. This makes the repository you are pushing to remember the set of branches that will be pushed out (e.g. if you always push maint and master there and no other branches, the repository you push to will have these two branches, and your local maint and master will be pushed there).

So I was using the default of simple meaning that it pushes the branch with the same name as the remote. Since I pushed to master, it pushed from my master to official/master.

In a way, I could see how some people might expect this behavior. In this situation, it was unexpected.

Instead, I want to change to use upstream to always push to the upstream branch. This can be changed with:

1
2
3
4
5


# Set for only the current repo
git config set --local push.default upstream

# Set for all repos
git config set --global push.default upstream

A COE on why technowizardry.net went down

Wed, 20 Aug 2025 21:23:00 +0000

COE = Correction of Error

My previous employer, Amazon, was a big proponent of doing blameless analysis of outages and figuring out what could be done to fix it. I recently had an outage on my servers and wanted to share what went wrong and the fix.

Summary

Starting Thursday until Friday, all TLS requests to a *.technowizardry.net domain would have failed due to a TLS certificate expiration error. Then on Friday, all DNS queries to a *.technowizardry.net zone failed which also caused mail delivery to fail too. This happened because cert-manager had created the acme-challenge TXT record, but the record was not visible to the Internet because the HE DNS was failing to perform an AXFR Zone Transfer from my authoritative DNS server. This was because PowerDNS was unable to bind to port :53 because systemd-resolved was already listening on that port.

While trying to fix the issue, I temporarily deleting the entire zone in Hurricane Electric which then triggered an issue how PowerDNS handles ALIAS records that blocked further AXFR transfers from succeeding. This required manual effort to unblock the transfer to let the zone start working again and allowed the TLS certificate to be renewed.

Additionally, Dovecot, which handles mail storage, failed to start after being rebooted because a Kubernetes sidecar container couldn’t start because of a “too many open files” error while setting up an inotify (to listen for file changes) because of a low ulimit.

Background

DNS

On each of my three servers, I run the PowerDNS authoritative server storing zone records in MySQL. I use cert-manager to create and renew TLS certificates from Let’s Encrypt in my Kubernetes cluster. Since I use wildcard certificates, Let’s Encrypt requires me to use the DNS verification method which means cert-manager has to have permissions to create and modify TXT records. It creates records in PowerDNS using the API.

I have three dedicated servers running my Kubernetes cluster. Three is chosen to ensure that if one machine is offline, I can continue to serve critical services, like this blog. However, they are located geographically all in the north-east area of the US and Canada. For performance, I wanted to distribute out the DNS servers so they’re faster. Thus, I adopted Hurricane Electric’s DNS service to serve as authoritative DNS servers. These servers automatically perform what’s called an DNS Zone Transfer or an AXFR from my servers to their servers to keep in sync.

When cert-manager updates a record using the API, PowerDNS automatically updates the SOA record to tell other secondary resolvers that there’s an update. As of now, my SOA record looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


$ dig +short SOA technowizardry.net
ns1.he.net. contact.technowizardry.net. 2025082101 3600 600 259200 60

ns1.he.net - Primary NS server
contact.technowizardry.net - A contact email
2025082101 - Serial number. Translates to date: "2025-08-21" version: "01"
3600 - The refresh time, how often secondary resolvers should try to refresh in seconds (1 hour)
600 - Time between retries
259200 - Expiration time. After 7 days, secondaries stop returning cached records
60 - Minimum TTL for the SOA

HE DNS checks for any changes to the zone serial number on my server, and if mine is newer, it pulls the newest version.

Mail

I use Dovecot to store email and implement IMAP/POP3. This is running in Kubernetes and is exposed as a Kubernetes service. It’s not configured in a Highly Available mode and only has one instance running because I never spent the time to figure out Dovecot HA. I then expose the port using ingress-nginx’s mechanism to expose TCP services.

Running as a side-car, I have a config-reloader container which runs quietly and listens for changes in the ConfigMap configuration files or the SSL certificate and if a change is detected, it sends a SIGHUP to Dovecot to reload the configuration. This allows me to avoid having to fully delete and restart the service to make config changes and automatically handle SSL certificate renewals. The deployment looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


apiVersion: apps/v1
kind: Deployment
metadata:
 name: dovecot
 namespace: mail
spec:
 template:
 spec:
 containers:
 - args:
 - '-c'
 - /etc/dovecot/dovecot.conf
 - '-F'
 command:
 - /usr/sbin/dovecot
 image: ajacques/kube-mail:latest
 name: dovecot
 resources:
 requests:
 memory: 64Mi
 volumeMounts:
 - mountPath: /etc/dovecot/conf.d
 name: config
 readOnly: true
 - env:
 - name: CONFIG_DIR
 value: /etc/dovecot/conf.d,/etc/dovecot/ssl/tls-combined.pem
 - name: PROCESS_NAME
 value: dovecot
 image: ajacques/config-reloader-sidecar:latest
 imagePullPolicy: IfNotPresent
 name: config-reloader-sidecar-config
 resources:
 limits:
 memory: 8Mi
 requests:
 cpu: 5m
 memory: 8Mi
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 add:
 - KILL
 drop:
 - ALL
 privileged: false
 readOnlyRootFilesystem: false
 runAsNonRoot: false
 volumeMounts:
 - mountPath: /etc/dovecot/conf.d
 name: config
 readOnly: true
 - mountPath: /etc/dovecot/ssl
 name: mailcert
 readOnly: true
 volumes:
 - configMap:
 defaultMode: 292
 name: dovecot
 optional: false
 name: config
 - name: mailcert
 secret:
 defaultMode: 292
 optional: false
 secretName: technowizardry-wildcard
 # ...

Investigation

DNS

I was on vacation with limited access to the Internet so it took a lot longer than it should have to figure it out and fix the problem.

First thing to check is server logs. cert-manager showed:

1

E0815 11:23:44.790373       1 sync.go:190] "propagation check failed" err="DNS record for \"technowizardry.net\" not yet propagated" logger="cert-manager.challenges" resource_name="technowizardry-prod-18-1227467484-534542642" resource_namespace="technowizardry" resource_kind="Challenge" resource_version="v1" dnsName="technowizardry.net" type="DNS-01"

The PowerDNS pods showed:

1
2
3
4
5
6
7


Aug 14 18:06:18 Guardian is launching an instance
Aug 14 18:06:18 Unable to bind UDP socket to '0.0.0.0:53': Address already in use
Aug 14 18:06:18 Fatal error: Unable to bind to UDP socket
Aug 14 18:06:19 Our pdns instance exited with code 1, respawning
Aug 14 18:06:20 Guardian is launching an instance
Aug 14 18:06:20 Unable to bind UDP socket to '0.0.0.0:53': Address already in use
Aug 14 18:06:20 Fatal error: Unable to bind to UDP socket

Comparing SOA records shows a difference. HE is not pulling updates.

1
2
3
4
5


dig +short SOA @srv5.technowizardry.net technowizardry.net | awk '{ print $3 }'
2025081401

dig +short SOA @srv5.technowizardry.net technowizardry.net | awk '{ print $3 }'
2025062601

Without SSH access and using an eSIM with a very limited data cap, I can’t investigate what’s listening, but it’s definitely the systemd-resolved. In the mean time, I use Hurricane Electric’s website to change from a secondary DNS zone type to a primary zone which means I can edit the records in the website and skip AXFR. A day later I had full Internet access and could get to work. I wanted to switch back to using the DNS Zone transfer mechanism and have Hurricane Electric pull from my PowerDNS, but they kept failing to pull my server. I couldn’t figure out why because the server was responding to some queries.

Manually trying to pull the AXFR zone didn’t succeed and gave no error messages back. I use TSIG to authenticate transfers. I double and triple-checked that it was correct.

1

$ dig -t AXFR -y hmac-sha512:[tsigkeyname]:[tsigkey]@144.217.181.222 technowizardry.net

The server logs gave no error messages as to why. In the pdns.conf, I changed the logging level from 3 to 5 to increase logging messages.

1
2
3
4


# pdns.conf
# loglevel Amount of logging. Higher is more. Do not set below 3

loglevel=5

Then retried and got this:

1

Aug 15 11:02:02 AXFR-out zone 'technowizardry.net', client '217.138.75.171', error resolving for ALIAS ingress-nginx.technowizardry.net., aborting AXFR

Ah ha. I’m using an ALIAS record type on apex record, technowizardry.net because I can’t use a CNAME record (CNAME tells resolvers to go look at another record to find the value) CNAMEs can’t be used on the root domain because of “reasons” that are boring, legacy, and unfortunate. I use these records because I point everything to a common record ingress-nginx.technowizardry.net which includes all three servers. As I do maintenance, I take them out and put them back in that single record.

Unfortunately, it seems that even though technowizardry.net is an ALIAS to ingress-nginx.technowizardry.net which this server knows about, it ends up trying to query a resolver on the Internet for that value. However, in this case, no server (which is my HE DNS) is able to handle that query because they’re failing to AXFR transfer. Thus we have a circular loop.

I consider this to be a bug in PowerDNS given that it can absolutely handle this query. However, I suspect they may not have implemented this to avoid issues where somebody creates an ALIAS on a subdomain that is on another NS server.

I quickly disable all ALIAS records to get back in business.

1
2
3


update records
set disabled = 1
where domain_id = 1 and type = 'ALIAS'

After that the DNS zone transfer succeeds. I can re-enable the ALIAS records. Eventually, the TLS certificate renewal succeeds and we’re mostly golden.

Mail

Except for mail is not coming back. The pod is marked as unhealthy which means that ingress-nginx won’t forward traffic to it. However, Dovecot itself is fine. It’s the config reloader that’s restarting with a “too many open files” error. It only opens like five files. How could that be. Well, it’s the only container that uses Inotify which is a Linux feature that sends notifications when files are updated. This one was interesting. Some GitHub issues talk about this, but it’s unclear. My intuition suggests it’s a ulimit or sysctl issue causing it to be unable to startup. Coincidentally, the pod is running on my node that is running NixOS. The other two haven’t been changed over yet.

1
2
3


sysctl -a | grep inotify
fs.inotify.max_user_instances = 128
user.max_inotify_instances = 128

The limit was pretty low. That’s the culprit.

Why

The next section is asking Five whys to get to the bottom. I’ll break it down to different problems.

Why didn’t cert-manager renew the certificate?

cert-manager was unable to renew certificates. Why?
cert-manager was waiting for the acme-challenge TXT record to be available? Why wasn’t it available?
cert-manager had created the TXT record in the authoritative PowerDNS instance, but the NS servers listed on the technowizardry.net zone (run by Hurricane Electric) did not have the TXT record. Why didn’t they have the record?
Hurricane Electric failed to perform a DNS zone transfer. Why couldn’t they transfer?
The DNS query was not making it to PowerDNS which could handle the transfer. Why?
PowerDNS wasn’t able to bind to the port because systemd-resolved was already listening. Why wasn’t it already disabled?
Not sure. Maybe an OS upgrade reverted my change or maybe I had only temp disabled it.

Why didn’t mail services start to work?

The dovecot service wasn’t starting up. Why?
Because the config-reloader side-car was crash looping. Why?
Because it was unable to setup the inotify that was needed. Why?
The sysctl was set to low. Why?
It was never changed from the default. Why?
First time I hit this problem

Action Items

Disable systemd-resolved resolver

The first action is to disable the systemd-resolved local resolver which listens on the same port. My servers then forwards DNS off host instead of using a local cache.

On NixOS, this can done using:

1
2
3
4
5
6


{ config, lib, pkgs, ... }:
{
 services.resolved.extraConfig = ''
 DNSStubListener=no
 '';
}

And on other machines, it’s like:

1
2
3
4


# /etc/systemd/resolved.conf.d/disable-local-resolver.conf

[Resolve]
DNSStubListener=no

Increase inotify limits

Next, we need to increase the sysctl limits. The Nix team discussed increasing this limit by default, but the related PR was abandoned.

1
2
3
4
5
6
7


{ config, lib, pkgs, ... }:
{
 boot.kernel.sysctl = {
 "fs.inotify.max_user_instances" = "8192";
 "user.max_inotify_instances" = "8192";
 };
}

Add more monitoring

I’m using kube-prometheus-stack to monitor my cluster. I didn’t know my certificate was failing to renew for over a week. Had I known, I could have fixed it ahead of the expiration. Let’s add an alert so I get an email next time.

Here I create a PrometheusRule object which tells Prometheus’ AlarmManager to start watching for this metric expression to trigger.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
 name: cert-manager-rules
 namespace: cert-manager
spec:
 groups:
 - name: certificates
 rules:
 - alert: failing-to-renew
 annotations:
 summary: >-
 Certificate {{ $labels.exported_namespace }}/{{ $labels.name }} is
 failing to renew
 expr: >-
 sum(certmanager_certificate_renewal_timestamp_seconds - time()) BY
 (name,exported_namespace) < 0
 for: 0s
 labels:
 severity: critical

I use Grafana dashboards to visualize and send alerts. I configured a contact point that sends me an email and a notification policy to catch the above alarm.

Over-engineering my Home Assistant HVAC Dashboard

Mon, 21 Jul 2025 23:47:00 -0700

Ever wondered how well your HVAC system is working in your home or condo? I did to an unhealthy degree. I want to know not just what’s the temperature, but how often is it running, what’s the supply and return temperatures, etc.? Let’s overengineer another project.

To start, I’ve got an Ecobee thermostat and use Home Assistant to integrate with all my devices. Home Assistant is an open-source smart-home orchestration system. It has numerous different integrations, which allow you to add almost any smart home device you can think of. I switched to after trying both Samsung SmartThings and Hubitat and finding of them not powerful.

Why?

With any thermostat added in Home Assistant, you can obviously know what is the temperature and humidity in different rooms and that’s useful, but HVAC units have a lot of different things that can be monitored.

For example, comparing the return temp (the temperature of the air going into the system) and the supply temp (air temp going back to your house) tells you how much the unit is able to increase or lower the temperature of the air.

Monitoring how long it runs per day helps me know whether it’s under sized or the setpoint is too low/high. I can even use this to know that I need to replace the filter after a certain number of hours of runtime.

My condo has a water sourced heat pump with a central water loop. When heating, it takes heat out of the loop and heats the air. When cooling, it takes heat in the air and transfers it to the water in the loop to go to the cooling tower up top. I attached a thermocouple to the pipe to measure that loop temp since it fluctuates based on how many other people are using it.

Home Assistant

First, add the thermostat to Home Assistant if it already isn’t already there. I’m using an Ecobee. If you have a different thermostat, you should still be able to use these dashboards, but you’ll need to ensure your thermostat provides similar sensors

Ecobee HomeKit

If you’ve got an Ecobee, then adding it using the Apple HomeKit Device integration will use local only network connections and avoid any cloud calls.

Ecobee Cloud Integration

There is also an Ecobee integration, but Ecobee stopped allowing people to sign up for the developer accounts. There’s a few features that the HomeKit integration doesn’t support like better mode switching because the HomeKit mode doesn’t set an override like the Ecobee integration does. I have both added, but disabled polling so I only make API calls when calling actions.

Basic Dashboards

Now, I’ve got some basic entities that report temperature, humidity, even motion. How you design your dashboard is up to your personal preference and the later metrics are more interesting. Don’t spend too much time here.

Motion

Ecobee thermostats and the smart reports report motion making them useful for motion tracking. A history graph makes it easy to see this across different rooms: And the YAML:

1
2
3
4
5
6
7
8
9


type: history-graph
entities:
 - entity: binary_sensor.home_motion
 name: Living Room
 - entity: binary_sensor.living_room_motion
 name: Office
 - entity: binary_sensor.bedroom_motion
 name: Bedroom
title: Motion

Thermostat Control

What dashboard isn’t complete without actually allowing us to edit the set points.

1
2
3
4


type: thermostat
entity: climate.home
features:
 - type: climate-hvac-modes

Calculating Runtime

Let’s get into the more complex metrics–runtime. Tracking runtime is a useful metric because it gives you some interesting insights:

Runtime - The more your A/C runs, the more wear it has and the more likely it’ll break down. You could use this to decide to schedule a yearly maintenance/inspection sooner or later depending on how much you use it.
Duty Cycle is the percentage of time that your unit is actually running. 100% means it’s running continuously. It could mean different things including: it’s a very hot/cold day, poor house insulation, improper setpoints, undersized HVAC units, or even a window left open.
Runtime since the filter was changed. As filters get dirtier, it gets harder to pull air through it making your system work harder. Tracking this gives you a metric to know when to replace your filter.

State Tracking

First, we’ll need to create a few template sensors to break out the states. You can either put this into your Home Assistant’s /config/configuration.yaml file, or create these using the UI in Settings > Devices & services > Helpers > Create Helper.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


template:
 - binary_sensor:
 # binary_sensor.hvac_fan_state
 # This tracks whether the fan is running
 # The fan will be running when it's in heat/cool or just running the fan
 # It tracks the unit actually working
 - name: "HVAC Fan State"
 icon: mdi:fan
 unique_id: 0k79upwzyOUladHiSg3R
 availability: "{{ has_value('climate.home') }}"
 state: >
 {{ (state_attr("climate.home", "hvac_action") in ["heating", "cooling", "fan"]) }}
 - sensor:
 # sensor.hvac_run_state
 # This tracks what the unit is doing, whether heating, cooling, or just spinning the fan
 - name: "HVAC Run State"
 unique_id: NtHeqUZpB2caZAA8PQyz
 availability: "{{ has_value('climate.home') }}"
 state: "{{ state_attr('climate.home', 'hvac_action') }}"

If you’re using YAML, save the file, then reload your entities in Developer tools >YAML tab > Reload “Template Entities” if it’s available, or restart HA if it’s not. You should now be able to see those new entities (Search in the top right if you don’t know where they are). If they’re marked as unavailable, then check to make sure your thermostat entity is climate.home. If it’s something else, like climate.my_ecobee, then update the templates above.

Calculating runtimes

Next, it’s time to use those templates to actually track the runtime. Home Assistants history stats integration is perfect for this.

These entities will start counting up whenever the fan state or run state is in the defined mode.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


sensor:
 - platform: history_stats
 name: HVAC Runtime Today
 entity_id: binary_sensor.hvac_fan_state
 state: 'on'
 type: time
 start: '{{ today_at() }}'
 end: '{{ now() }}'
 - platform: history_stats
 name: HVAC Runtime Cooling Today
 entity_id: binary_sensor.hvac_run_state
 state: 'cooling'
 type: time
 start: '{{ today_at() }}'
 end: '{{ now() }}'
 - platform: history_stats
 name: HVAC Runtime Heating Today
 entity_id: binary_sensor.hvac_run_state
 state: 'heating'
 type: time
 start: '{{ today_at() }}'
 end: '{{ now() }}'

Filter Change Tracking

The sensor.hvac_runtime_today resets every day, so if we want to track total runtime since a filter was replaced weeks or months ago, we need to keep summing it up.. The utility meter integration can do exactly that. I also store the filter change time just to make it easier.

In /config/configuration.yaml, create the following entities and reload.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


input_datetime:
 # Tracks when the filter was replaced
 hvac_filter_change_date:
 name: HVAC Filter Change Date
 has_date: true
 has_time: true

utility_meter:
 energy:
 name: HVAC Runtime Since Filter Change
 unique_id: SmRwiJmHh4FdbVftTDPD
 source: sensor.hvac_runtime_today

Dashboarding

Now, I’ve got a few different entities, here’s some sample dashboards. If you don’t already have it, I use the template-entity-row dashboard card row to help with some more complicated dashboards.

And the YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


type: entities
entities:
 - entity: binary_sensor.hvac_fan_state
 - entity: sensor.hvac_run_state
 - entity: sensor.hvac_runtime_since_filter_change
 - entity: sensor.hvac_runtime_today
 - entity: sensor.hvac_cooling_runtime_today
 - entity: input_datetime.hvac_filter_change_date
 type: custom:template-entity-row
 name: Filter last changed
 state: >
 {{ time_since(states('input_datetime.hvac_filter_change_date') |
 as_datetime) }} ago
title: HVAC

Entity Reference

Let’s recap the entities we have created.

input_datetime.hvac_filter_change_date - Tracks the last time the filter has been changed
sensor.hvac_runtime_since_filter_change - Tracks how many minutes the unit has operated since changing the filter
binary_sensor.hvac_fan_state - Tracks whether the fan is running
sensor.hvac_run_state - Tracks what the thermostat is calling for (heating, cooling, fan, etc.)
sensor.hvac_runtime_today - Tracks how much the unit has run today
sensor.hvac_cooling_runtime_today - Tracks how much cooling the unit has done today
sensor.hvac_heating_runtime_today - Tracks how much heating the unit has done today

Sensing Temperatures

In this post, I showed some basic dashboarding for a thermostat in Home Assistant and also showed some advanced monitoring that used Home Assistant to track running times and temperatures for your HVAC unit which gives useful insights for maintenance and efficiency. The new metrics are easy to create and add to dashboards.

Going a step further, I created my own embedded device to measure air temperatures. Next up, I wanted to add some extra sensors to my HVAC unit itself. My system is self-contained inside my condo and I have access to the different pipes and vents, but a split unit installed outside may not be as accessible.

To handle this, I built my own prototype circuit. I started with an ESP32 running ESPHome. You can use any type of ESP32, but I had a few of the Adafruit Qt Py ESP32s lying around. ESPHome is a firmware development system that abstracts out writing C++ and you define sensors in YAML.

For my sensors, I opted for these thermocouples because they’re easy connect to the board and place them in specific areas. Thermocouples work by measuring change in resistance as the temperature changes. They don’t have any electronics themselves and require a separate component, an amplifier to convert that into something that the ESP32 can read.

Adafruit sells several different types of amplifiers including the MCP9600, MCP9601, MAX31856, and the MAX31850K, and a few others.

The MCP9601 is nice because it comes with a stemma connector, which is an Adafruit standard connector that allows for you to connect i2c devices without soldering, but the issue with i2c is that the address is hard-coded to a single address. Multiple amplifiers will conflict unless you change the address by adding a small amount of solder to the address change pads below.

The MAX variants are less accurate (0.25°C resolution) compared to the MCP960x at 0.0625°C resolution.

My first revision opted for the MCP9600 and I soldered the pins to the circuit and put them into a breadboard.

One thermocouple was inserted as close as possible to the supply duct that supplies the rest of the condo with heated or cooled air. The 5 meter thermocouple makes it easy to run the temp sensor right into it. The thermocouple wire can be seen in the picture below along the right side of the unit.

I also secured one to the water loop supplying hot and chilled water to my unit. During heating modes, it allows me to see how much heat I could put into the air. During cooling modes, the hotter it is, the harder my unit will work to put heat into it. This also gives me some data I can use to work with my building’s engineer to optimize the systme.

ESPHome Config

Here’s the configuration I used. If you use a different ESP32 board, some of this configuration will differ.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


esphome:
 name: qt-py-hvac-sensor
 platformio_options:
 board_build.f_cpu: 160000000L # 160MHz
 board_build.f_flash: 40000000L
 board_build.flash_mode: dio
 board_build.flash_size: 4MB


esp32:
 board: adafruit_qtpy_esp32
 variant: ESP32
 framework:
 type: arduino
 version: 2.0.6

wifi:
 ssid: !secret wifi_ssid
 power_save_mode: none
 password: !secret wifi_password

substitutions:
 device: "Qt Py Heat Pump"

mqtt:
 broker: mqtt.home.ajacqu.es
 discovery_unique_id_generator: mac
 discovery_object_id_generator: device_name

ota:
 - platform: esphome
 password: "..." #changeme

i2c:
 - id: stemma
 sda: SDA1
 scl: SCL1
 scan: true
 frequency: 10kHz

sensor:
 - platform: mcp9600
 id: temp
 hot_junction:
 name: "Loop Water Temperature"
 retain: false
 thermocouple_type: K
 address: 0x67
 update_interval: 60s

 # On a 5 meter cable to the output manifold
 - platform: mcp9600
 id: temp2
 hot_junction:
 name: "Air Supply Temperature"
 retain: false
 thermocouple_type: K
 address: 0x60
 update_interval: 60s

Future Work

In my next iteration, I plan to integrate a 24v DC regulator to pull power directly from the HVAC unit instead of having to run an ugly DC wire to a nearby closet. I want to add a few more thermocouple amplifiers to measure other areas, or even measure the pressure drop across the fan like this project, esphome-pressure demonstrates. I also want to make it a custom PCB so the board is a lot cleaner.

Conclusion

Going a step further, I created my own embedded device to measure air temperatures.

Abandon the Helm, leveraging CDK for Kubernetes

Mon, 14 Apr 2025 18:27:00 -0700

I’ve had enough of Helm. I don’t know who thought string-based templating engines would be a good idea, but I have had one too many indention relate bugs. They’re a source of bug and a pain. Kubernetes YAML files just contain a ton of boiler-plate YAML configuration. Like how many times do I have to specify the labels? Its spec/template/spec for Deployment, but spec/jobTemplate/spec for CronJob. Ain’t nobody got time to remember that.

Enter cdk8s. It’s built-upon CDK, a software development kit that uses standard programming languages, like TypeScript, Python, or Java, as a way to define resources that then get compiled into YAML or JSON to upload to CloudFormation, or in our case, Kubernetes.

Why would you want/need a full programming language just to define some infrastructure? Well, there are some benefits. Let’s go through them.

The Good

No more string based templating

In Helm, when you’re templating files, you use Golang’s text templating system. You start writing YAML text, then depending on your use case, mix in some variables, some conditionals, some loops, and more. At first, it seems reasonable and maybe you’ve only got a few variables to swap out (snippet source):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


apiVersion: apps/v1
kind: Deployment
metadata:
 name: web
 labels:
 app: {{ template "sentry.name" . }}
 chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
 release: {{ .Release.Name }}
 heritage: {{ .Release.Service }}
# ...
 metadata:
 annotations:
 checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
 checksum/config2: {{ include (print $.Template.BasePath "/sentry-config-file.yaml") . | sha256sum }}
 checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
 labels:
 component: web
 chart: {{ .Chart.Name }}
 spec:
 containers:
 - args:
 - run
 - web
 {{- if .Values.ingress.tls_secret_name }}
 env:
 - name: SENTRY_USE_SSL
 value: "1"
 {{- end }}

Maybe it stays that way, or maybe you need more and more variables, substitutions. Especially if you’re vending a Helm chart to others. Then you end up with everything needed to be passed as values like this template from ingress-nginx.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


apiVersion: apps/v1
kind: DaemonSet
metadata:
 labels:
 {{- include "ingress-nginx.labels" . | nindent 4 }}
 app.kubernetes.io/component: controller
 {{- with .Values.controller.labels }}
 {{- toYaml . | nindent 4 }}
 {{- end }}
 name: {{ include "ingress-nginx.controller.fullname" . }}
 namespace: {{ include "ingress-nginx.namespace" . }}
 {{- if .Values.controller.annotations }}
 annotations: {{ toYaml .Values.controller.annotations | nindent 4 }}
 {{- end }}
spec:
 selector:
 matchLabels:
 {{- include "ingress-nginx.selectorLabels" . | nindent 6 }}
 app.kubernetes.io/component: controller
 revisionHistoryLimit: {{ .Values.revisionHistoryLimit }}
 {{- if .Values.controller.updateStrategy }}
 updateStrategy: {{ toYaml .Values.controller.updateStrategy | nindent 4 }}
 {{- end }}
 minReadySeconds: {{ .Values.controller.minReadySeconds }}
 template:
 metadata:
 {{- if .Values.controller.podAnnotations }}
 annotations:
 {{- range $key, $value := .Values.controller.podAnnotations }}
 {{ $key }}: {{ $value | quote }}
 {{- end }}
 {{- end }}
# ...

By that point, you’ve lost all meaning of templating. The YAML has become but a shell of its original self, it’s merely just a vessel that the values.yaml get passed through onto their final destination that is your Kubernetes cluster. Then you ask yourself, is this the best it can be? Is any of this logic correct? Can you even tell at a glance without unit tests? Wait, a templating system has unit tests? Indeed.

Now, you get to define and expose class based properties and fields. For example, instead of explicitly listing every single property that can be overridden, you can do:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


// Allow user to override a few props
type OverridableProps = Pick<DaemonSetProps, 'image' | 'imagePullPolicy'>;

function makeDaemonSet(scope: Construct, props: OverrideableProps)
 new DaemonSet(app, 'nginx', {
 name: 'controller',
 image: 'registry.k8s.io/ingress-nginx/controller',
 // Allow the user to override things:
 ...props
 });
}

No more indention hell

Using Helm charts means that you’re forced to be very careful about indention. It gets worse once you start mixing templates into the picture.

For example, I found this bug in ingress-nginx caused by improper indention:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


 initContainers:
 - command:
 - /bin/chown
 - "101"
 - /mycache
 image: busybox:latest
 name: chmod
 volumeMounts:
 - mountPath: /mycache
 name: cache
 # <--- Notice that this is aligned under the volumeMounts, not under initContainers
 # <-- Also note the empty image and names
 - name:
 image:
 command: ['sh', '-c', '/usr/local/bin/init_module.sh']
 volumeMounts:
 - name: modules
 mountPath: /modules_mount

The bug was because the following snippet had one too many tabs at the beginning, but you’d never figure know that just looking at it:

1
2
3
4
5


 {{- if .Values.controller.extraModules }}
 {{- range .Values.controller.extraModules }}
 - name: {{ .Name }}
 image: {{ .Image }}
 command: ['sh', '-c', '/usr/local/bin/init_module.sh']

CDK fixes that. You no-longer care about indention, unless of course you use the Python bindings for CDK8s. You use normal objects and set properties on them, call methods, etc. CDK8s then is responsible for serializing that into a properly indented YAML file.

Reusable functions

With Helm templating, you frequently end up just having a lot of boiler-plate YAML tags around. For example, a bunch of my network policies ended up having the same egress policy in them. Before, I would copy and paste the following file into many different namespaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: ...
 namespace: ...
spec:
 egress:
 # Allow access to DNS
 - ports:
 - port: 53
 protocol: UDP
 - port: 53
 protocol: TCP
 to:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: kube-system
 podSelector:
 matchLabels:
 k8s-app: kube-dns

Helm does support a reusable template, but I was using Rancher Fleet which I don’t know if it supports them and they suffer from the indention problem mentioned above.

With code-based solutions, this entire model changes because I can just write a method that can be re-used by different constructs. This one of the most powerful features of code-based infra as code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


function grantDns(context: Construct, netPolicy: kplus.NetworkPolicy) {
 const peer = kplus.Pods.select(context, 'netpol-kube-dns', {
 namespaces: kplus.Namespaces.select(context, 'netpol-kube-system', {
 names: ['kube-system']
 }),
 labels: {
 'kube-app': 'kube-dns'
 }
 });

 netPolicy.addEgressRule(peer, [
 NetworkPolicyPort.tcp(53),
 NetworkPolicyPort.udp(53)
 ]);
}

const netPol = new kplus.NetworkPolicy(context, 'netpol');

grantDns(context, netPol);

Compile time type-safety

Does that property exist on that resource? Are you missing anything critical? Any invalid field values? YAML provides no compile-time validation. I like to use yamllint to see if it’s syntactically valid YAML, but that doesn’t validate that fields exist.

CDK8s gets this right. Fields that don’t exist on the resource can’t be set in code and it simply does not compile.

The Bad

Can’t move resources

This is another problem with the underlying CDK design. In CDK, resource names are derived from the path of constructs and resource names. A resource might have the path App/Chart/MyService/Deployment, which has the resource name chart-myservice-deployment-c873a441.

If I try to rename the construct name to Deployment2, I get a new resource name Chart-MyService-Deployment2-c8cb06b1 and helm will delete the old one and create a new deployment.

This is dangerous. When you’re writing code, you easily forget about this because sometimes a code refactoring is needed to fix some issue, but you can’t always safely do this because Helm will end up deleting and recreating a resource. Helm also doesn’t have CloudFormation’s safe deployment mechanism where dependencies between resources are identified and creates are deployed first, then if everything succeeds, deletes are performed. ==Helm has a static defined order on how it deploys resources==. This sort of replicates CloudFormation’s deployment ordering strategy, but it doesn’t actually guarantee that resources are working correctly. For example, a PersistentVolumeClaim can be created, but fail to actually provision and the deployment will get stuck.

Deployment tooling

CDK8s itself provides a bunch of classes that allow you to generate YAML files, but those are not useful unless you have something that actually deploys those files. This tooling for deployments is very important.

It is the glue that takes the synthesized output files and actually gets them onto your K8s cluster. Base CDK does pretty well with this because it inherently only has to work with AWS and has a built-in CLI to deploy to CloudFormation or using CDK you can deploy with CodePipelines or GitHub Actions. To be honest, I only worked with the Amazon-internal variant of the pipeline which is different and works well.

What are we missing? The CDK CLI itself provides a command cdk deploy that synthesizes your output, identifies the dependencies between the stacks, then sequentially deploys the stacks in transitive order using CloudFormation.

I want the thing that manages the deployment part of CI/CD. Kubernetes has a lot of options here. Let’s explore some

What about Helm?

Helm provides a CLI that given a YAML manifest, it will create, update, and delete the resources in Kubernetes. I could use GitHub Actions to first synthesize my CDK8s application, then deploy it like below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


jobs:
 validate:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v3
 - id: synth
 name: Synth cdk8s manifests
 run: |-
 npm install
 npm run build

 - name: Set up Kubectl
 uses: azure/setup-kubectl@v4.0.0
 with:
 version: 'v1.31.0'

 - name: Deploy to cluster
 uses: azure/setup-helm@v4.3.0

 - name: Create kubeconfig file
 run: echo "${{ secrets.KUBECONFIG }}" > ${{ github.workspace }}/kubeconfig

 - name: Deploy to Kubernetes using Helm
 if: github.ref == 'refs/heads/master'
 env:
 KUBECONFIG: ${{ github.workspace }}/kubeconfig
 run: |
 helm upgrade --install mytarget dist/ --namespace default

Functionally that works, however it lack some features:

Can’t deploy to multiple Helm releases. My Git repo has several different Helm charts and releases that got deployed. Some of them had to be deployed first. CDK8s doesn’t make this easy.

1
2
3
4
5
6
7


const app = new cdk8s.App();

const chart1 = new cdk8s.Chart(app, 'common');
const chart2 = new cdk8s.Chart(app, 'service1');
const chart3 = new cdk8s.Chart(app, 'service2');

app.synth();

Everything gets emitted to the dist folder. Because there’s only one Chart.yaml and everything is under the same templates, I can’t use Helm CLI to deploy one file to one release.

1
2
3
4


dist/Chart.yaml
dist/templates/common.k8s.yaml
dist/templates/service1.k8s.yaml
dist/templates/service2.k8s.yaml

If we look back to my GitHub Actions attempt:

1
2
3
4
5
6


- name: Deploy to Kubernetes using Helm
 if: github.ref == 'refs/heads/master'
 env:
 KUBECONFIG: ${{ github.workspace }}/kubeconfig
 run: |
 helm upgrade --install mytarget dist/ --namespace default

The only idea that I have to fix this is to emit separate chart files, then run a GitHub action that moves them all into separate Chart folders that Helm can deploy separately.

What about ArgoCD?

There’s also ArgoCD which is a common Kubernetes-native solution for managing deployments. I’ve avoided it thus far because it’s always seemed overly-complex with several different controllers all running for my use-case of just deploying some YAML to a cluster. Do I really want to need how many controllers just to deploy some YAML?

Also, it seems like when ArgoCD works with a Git repo, it doesn’t know how to first compile the manifests. At least, that’s what this blog post implies and it required two separate Git repositories.

Admittedly, I didn’t test out ArgoCD, so there might be something I’m missing, but I it still won’t fix some of the other issues when coding with YAML.

Working with legacy resources

If you have any existing resources written using raw YAML and are already created in an existing Helm release and you want to adopt cdk8s, you’re going to be in a tricky place. If you want to switch an existing Helm release from raw YAML to cdk8s, you have to either:

Synth the output and put them in the same folder as your legacy YAML

Whether this works or not depends on what kind of CI/CD you currently use. If you’re using something like Rancher Fleet, like I was until I realized it was fragile and frequently broke, then you now have to have two different Git repos, one with cdk8s, and one with raw YAML that uses your CI/CD to commit the output into that repo. [This post](If you’re using something like Racher Fleet, like I was, then you now have to have two different Git repos, one with cdk8s) talks about that model using GitHub Actions, but that complexity terrified me.

Import the legacy resources as-is and include them in the output

When I worked at AWS and owned a program to switch from an internal framework that used raw YAML CloudFormation to native CDK, we used CDK’s CfnInclude construct to do this. Cdk8s has an equivalent called Include

Messy auto generated resources

CDK8s has a number of dev-friendly methods, like the ability to create NetworkPolicies with permissions using just one line of code:

1

myService.connections.allowTo(redis.workload);

From a dev perspective, this is massive productivity boost. It can auto create both the ingress and egress policies. However, cdk8s generates multiple NetworkPolicies for each and every grant with automatically generated names.

From a debugging and operational perspective, this makes it difficult when you’re viewing your cluster resources and trying to figure out what policy applies and what each means.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allowegressc8116eb74577cb8dbda1910afbbd7c3456-c8832897
 namespace: foo
spec:
 egress:
 # ...
 podSelector:
 matchLabels:
 cdk8s.io/metadata.addr: example-TestService-Deployment-c8d415ef
 policyTypes:
 - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allowingresskube-systemc8116eb74577cb8dbda191-c803ee19
 namespace: kube-system
spec:
 ingress:
 # ...
 podSelector:
 matchLabels:
 kube-app: kube-dns
 policyTypes:
 - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allowegressc8b9fe52de2665a49bb6866db05c076914-c8924060
 namespace: foo
spec:
 egress:
 # ...
 podSelector:
 matchLabels:
 cdk8s.io/metadata.addr: example-TestService-Deployment-c8d415ef
 policyTypes:
 - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allowingressfooc8b9fe52de2665a49bb6866d-c8819c21
 namespace: foo
spec:
 ingress:
 # ...
 podSelector:
 matchLabels:
 cdk8s.io/metadata.addr: example-TestService-Redis-c8b9fe52
 policyTypes:
 - Ingress

Difference in mental model compared to YAML

I expect differences when comparing a programming language vs a YAML templates because they’re just fundamentally different styles of writing. YAML is the language for describing Kubernetes resources. Tutorials use them, this blog uses YAML for Kubernetes. With so much documentation using YAML, you now have to mentally translate that into the CDK8s equivalent. Generally they look pretty similar, like a container looks the same:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: homeassistant
spec:
 podSelector:
 matchLabels:
 app: homeassistant
 policyTypes:
 - Egress
 egress:
 # Allow DNS queries
 - ports:
 - port: 53
 protocol: TCP
 - port: 53
 protocol: UDP
 to:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: kube-system
 podSelector:
 matchLabels:
 k8s-app: kube-dns

And the equivalent in CDK8s:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


const netPolicy = kplus.NetworkPolicy(context, 'netpolicy');
const peer = kplus.Pods.select(context, 'netpol-kube-dns', {
 namespaces: kplus.Namespaces.select(context, 'netpol-kube-system', {
 names: ['kube-system']
 }),
 labels: {
 'kube-app': 'kube-dns'
 }
});

netPolicy.addEgressRule(peer, [
 NetworkPolicyPort.tcp(53),
 NetworkPolicyPort.udp(53)
]);

At first I didn’t even know about the Pods.select and Namespaces.select and assumed I had to create my own custom peer class, but it wasn’t until I started preparing to make a PR that I found this doc that explained this.

This is not unique to CDK8s, and is prevalent in CDK vs CloudFormation too or even Terraform/OpenTofu. It’s also not necessary a really bad problem, but the more field names start to differ. The more you have to reach for different classes or start to pass around CDK contexts and names, it just gets confusing.

Sure, I know the answer now, but these things will be confusing for the next person.

Unexpected defaults

CDK8s provides defaults for Deployments that I don’t think should be provided. For example, they provide a default resource request and a default security context.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


resources:
 limits:
 cpu: 1500m
 memory: 2048Mi
 requests:
 cpu: 1000m
 memory: 512Mi
securityContext:
 allowPrivilegeEscalation: false
 privileged: false
 readOnlyRootFilesystem: true
 runAsNonRoot: true

CPU resource limits are bad. You probably shouldn’t use them unless you know your process has a max thread count. On-top of that, CDK8s limits to 1.5 CPU cores which is not rounded and if you do have two threads running a lot, one of them is going to get throttled 50% of every second. I imagine they’re just picking some default value to help Kubernetes bin-pack, but in this case, I say you’re more likely just picking a bad number.

While enforcing read-only root file-systems is good security practice™, it’s also likely to cause a lot of broken software. So many software components I run require the ability to write temp files, etc. I’m not sure why they chose this default. Maybe they wanted to do opt-out security, which I mean if the software can run, great, but also it’s very inconsistent.

My opinion: CDK8s should use the same defaults as Kubernetes itself. If they want to provide secure, robust solutions, provide a higher level construct. I’ve seen this employed inside of Amazon for security sensitive CDK.

Conclusion

CDK8s provides some pretty useful dev productivity improvements over writing raw YAML defined resources. The built-in compile-time type-safety, code completion when in an IDE, and ability to abstract out repetitive code is a time saver.

However, it does have some down-sides. The big one is lack of the last bit of deployment tooling that actually helps this get deployed to my Kubernetes cluster.

This blog is now on ASP.NET Core

Sun, 23 Mar 2025 18:17:00 -0700

This blog is a static website compiled using Hugo. Up to this point, I built the website and packaged all of the assets into a Docker container with NGINX which was hosted on my dedicated server cluster.

This worked well and was simple, but I have an upcoming project that I’ll be announcing soon that required dynamic content that nginx + pure static files wasn’t easily able to implement with NGINX.

To fix this, I decided to migrate this blog from NGINX to ASP.NET Core. Here’s how and why.

Background

I’m still using Hugo to generate all HTML files in my blog from the raw Markdown files just like any other static website. I make changes, commit them to GitHub, then GitHub Actions runs hugo, builds the .net application, and packs the web assets into the image.

Why

Why C#? I like C# and .net and wanted to see how .net has evolved over since I last used it.

Why replace NGINX? My next project, to implement ActivityPub, required some dynamic features that were difficult to handle in NGINX. For example, I need to send push notifications on new posts and process inbox messages, NGINX can’t do that. I experimented with using the nginx-lua-module and implement the logic in Lua, but my use case was more complex. I also looked at Varnish which supported more imperative style response handling, but also lacked some features. Ultimately, I had a prototype that used NGINX, but the resulting architecture was more complex.

Static Files

First thing we do is serve files located in the /app/wwwroot folder (this is where we’ll put the Hugo output files.)

1
2
3
4
5
6
7


var app = builder.Build();
// ...

app.UseDefaultFiles(); // When a user requests /foobar/, serve up /foobar/index.html
app.UseStaticFiles();

app.Run();

The Dockerfile looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


FROM hugomods/hugo:0.123.0 AS hugobuild

COPY . /src
RUN HUGO_ENVIRONMENT=production hugo --minify -b https://www.technowizardry.net

FROM mcr.microsoft.com/dotnet/runtime:8.0 AS base
USER app
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
RUN apt-get update \
 && apt-get install -y --no-install-recommends \
 clang zlib1g-dev
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY ["http-server/http-server.csproj", "."]
RUN dotnet restore "./http-server.csproj"
COPY activity-publisher/ .
WORKDIR "/src/."
RUN dotnet build "./http-server.csproj" -c $BUILD_CONFIGURATION -o /app/build

# This stage is used to build the AOT-compiled output
FROM build AS publish
ARG BUILD_CONFIGURATION=Release
RUN dotnet publish "./http-server.csproj" -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=true

FROM ${FINAL_BASE_IMAGE:-mcr.microsoft.com/dotnet/runtime-deps:8.0} AS final
WORKDIR /app
COPY --from=publish /app/publish /app/
COPY --from=hugobuild /src/public /app/wwwroot/
ENTRYPOINT ["./http-server"]

Compressing static files

That’s easy, but the next problem is supporting compression. With NGINX, I can pre-compress files in the Docker image, then use the gzip_static option to serve that to clients. ASP.NET Core does support response compression, but the response gets compressed dynamically for every request, instead of using pre-compressed files. Compressing the file every time a client fetches it even though it never changes is wasted effort. Let’s do better.

I will support both gzip, which is the most common compression encoding, and brotli, a newer compression encoding that better compresses files.

Generating the files

First thing, we need to generate the pre-compressed files in the Dockerfile so they can be found. The command I used below compresses anything over 1KiB because smaller files often times don’t compress enough to be worth the overhead. I also exclude media files because they already have their own compression applied that won’t benefit from extra compression.

Here’s what the Dockerfile looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


FROM hugomods/hugo:0.123.0 AS hugobuild

COPY . /src
RUN HUGO_ENVIRONMENT=production hugo --minify -b https://www.technowizardry.net

# First, compress using gzip
RUN find /src/public ! -name '*.png' ! -name '*.jpg' ! -name "*.mp4" -size +1k -type f -print -exec gzip -k -f "{}" \;

# Then compress using brotli
FROM alpine:3 AS brotlibuild
RUN apk update && apk add --upgrade brotli
COPY --from=hugobuild /src/public /src/public/
RUN find /src/public ! -name "*.gz" ! -name "*.png" ! -name "*.jpg" ! -name "*.mp4" -size +1k -type f -print -exec brotli "{}" \;

# ...
COPY --from=brotlibuild /src/public /app/wwwroot/
ENTRYPOINT ["./http-server"]

Serving the files

Next, we need to look at every inbound HTTP request and see if the client passed in an Accept-Encoding header (MDN Web Docs). If specified, one or more values can be passed, but we’re interested in gzip, br, and identity.

This check will be added to the ASP.NET request processing chain. ASP.NET came with a class (DefaultFilesMiddleware.cs) that implemented part of what I wanted, but it didn’t support checking for .gz or .br pre-generated files.

Let’s break this down. The first chunk is just initialization. The _next delegate represents the next middleware in the chain. In this case, it’ll be the StaticFilesMiddleware. The _fileProvider provides a handle to the directory that contains our static files. When running in Docker, it’ll be /app/wwwroot.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


using Microsoft.Extensions.FileProviders;
using Microsoft.Extensions.Options;

public class CustomDefaultFileMiddleware
{
 private readonly RequestDelegate _next;
 private readonly IFileProvider _fileProvider;

 public CustomDefaultFileMiddleware(RequestDelegate next, IWebHostEnvironment hostingEnv, IOptions<StaticFileOptions> options)
 {
 // next middleware in the chain
 _next = next;
 // The file provider knows where to find the source files
 _fileProvider = options.Value.FileProvider ?? hostingEnv.ContentRootFileProvider;
 }

Next, this method is called for every request. Only GET/HEAD requests should be handled. POSTs/PUTs, etc. should never touch a static file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


 public Task Invoke(HttpContext context)
 {
 if (context.GetEndpoint()?.RequestDelegate is null
 && (context.Request.Method == "GET" || context.Request.Method == "HEAD"))
 {
 var subpath = context.Request.Path;

 // Add a slash at the end if it doesn't exist
 // This ensures that when we append the file name, it's a valid path
 // TODO: Consider doing a client redirect here instead to ensure
 // the user is always on a consistent location
 if (!subpath.Value!.EndsWith("/"))
 {
 subpath += new PathString("/");
 }

Next, we check to see if the folder exists. If the client passes in an Accept-Encoding header, we scan to see if we support any of the encodings and have a pre-compressed file on disk. We then set the path and, if applicable, a response Content-Encoding header and pass it along to the next middleware to handle.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


 var dirContents = _fileProvider.GetDirectoryContents(subpath.Value!);
 if (dirContents.Exists)
 {
 var acceptEncoding = context.Request.Headers.AcceptEncoding;
 foreach (var encoding in acceptEncoding)
 {
 // We use StartsWith because request encodings can include a ";q=0.5" to specify quality
 // but we ignore that
 if (encoding.StartsWith("br") && FileExists(context, subpath.Value!, fileToFind + ".br"))
 {
 fileToFind += ".br";
 context.Response.Headers.ContentEncoding = "br";
 break;
 }
 else if (encoding.StartsWith("gzip") && FileExists(context, subpath.Value!, fileToFind + ".gz"))
 {
 fileToFind += ".gz";
 context.Response.Headers.ContentEncoding = "gzip";
 break;
 }
 }
 // Match found, re-write the url. A later middleware will actually serve the file.
 context.Request.Path = new PathString(subpath + fileToFind);
 }
 }

 return _next(context);
 }
}

Then replace the previously used app.UseDefaultFiles, with our custom middleware:

1
2
3
4
5
6
7


var app = builder.Build();

// app.UseDefaultFiles()
app.UseMiddleware<CustomDefaultFileMiddleware>();
app.UseStaticFiles();

app.Run();

Response headers

But wait, there’s a problem. Opening up my website in a web browser causes me to download the page instead of viewing it. Looking at the response, the Content-Type response header shows a gzip MIME type vs HTML.

1
2
3
4
5
6
7
8


GET / HTTP/1.1
Host: localhost:1313
Accept-Encoding: gzip, deflate, br

HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 4499
Content-Type: application/x-gzip

This is happening because our CustomDefaultFileMiddleware says “hey serve up the index.html.gz file please” and the StaticFileMiddleware says “okay here we go, btw the mime type for .gz is application/x-gzip.” The correct HTTP implementation is to pass the MIME type as text/html, the MIME type of the un-encoded file.

This can be done by adding an OnPrepareResponse on the UseStaticFiles step that corrects the response MIME type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


// app.UseStaticFiles()
var contentTypeProvider = new FileExtensionContentTypeProvider();
app.UseStaticFiles(new StaticFileOptions
{
 ContentTypeProvider = contentTypeProvider,
 OnPrepareResponse = (context) => {
 if (context.Context.Response.Headers.ContentEncoding.Count > 0)
 {
 var subGz = context.Context.Request.Path.Value[..^3];
 if (contentTypeProvider.TryGetContentType(subGz, out var contentType))
 {
 context.Context.Response.Headers.ContentType = contentType;
 }
 }
 }
});

Now, the response Content-Type header shows the MIME type of the file being served and everything works.

1
2
3
4
5
6
7
8


GET / HTTP/1.1
Host: localhost:1313
Accept-Encoding: gzip, deflate, br

HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 4499
Content-Type: text/html

Cache-Control

Next up, we need to allow client caching. By default, no caching headers are being set to the client, so nothing is being cached and the client has to make a request to the server for every single file. Caching increases browsing performance and is controlled through the Cache-Control header (MDN docs).

In my blog, I only set caching headers for static assets that are hashed, like my CSS, JS, and images. Other content, such as HTML, will not be cached to ensure clients revalidate and see if there’s any new content.

To make this happen, we need yet another middleware handler. We’ll add this to the OnPrepareResponse

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


private readonly string MAX_CACHE_HEADER = new CacheControlHeaderValue()
 {
 Public = true,
 MaxAge = TimeSpan.FromDays(365 * 10) // 10 years
 }.ToString();

// ...

var contentTypeProvider = new FileExtensionContentTypeProvider();
app.UseStaticFiles(new StaticFileOptions
 {
 ContentTypeProvider = contentTypeProvider,
 OnPrepareResponse = (context) => {
 var response = context.Context.Response;
 var workingPath = context.Context.Request.Path.Value;
 if (response.Headers.ContentEncoding.Count > 0)
 {
 workingPath = workingPath[..^3];
 if (contentTypeProvider.TryGetContentType(workingPath, out var contentType))
 {
 response.Headers.ContentType = contentType;
 }
 }
 if ( urlPath.StartsWithSegments("/scss")
 || urlPath.StartsWithSegments("/ts")
 // Don't cache SVGs because Hugo doesn't hash the file name
 // If we fix that, we can check the mime type
 || workingPath[^4..] == ".png")
 {
 response.Headers.CacheControl = MAX_CACHE_HEADER;
 }
 }
 });

With that, we now have caching headers set and clients won’t refetch static content.

1
2
3
4
5
6
7
8


GET /ts/main.abc.js HTTP/1.1
Host: localhost:1313
Accept-Encoding: gzip, deflate, br

HTTP/1.1 200 OK
Content-Length: 7835
Content-Type: text/javascript
Cache-Control: public, max-age=315360000

Conclusion

With these changes, I now have a fully equivalent replacement for my previous NGINX web server. I had to implement code to select which default file to return when a user requests /foobar/ and support serving pre-compressed static files which required code to fix MIME types. This was trivially handled by NGINX with the gzip_static on directive. I also had to add cache control headers, also handled in one line with NGINX’s expires max directive.

Looking at this comparison in isolation, I ended up with more code compared to what NGINX already handles out of the box. However, I did this not just for the sake of change, but because when I implemented ActivityPub, I built on-top of this server and implemented features that were not easy with NGINX.

Fixing common Hugo encoding problems

Sat, 15 Mar 2025 21:10:00 -0700

I posted a link to my blog on Slack and was greeted with HTML entities right in the website summary. I could see certain characters like the apostrophe ’ being encoded as ’.

Here’s how I fixed this problem.

HTML Meta Tags

Investigating

Slack and social media sites use meta tags defined by the OpenGraph protocol to fetch information like the summary, publish dates, and images relevant for posting on a feed. On this post, we can see those tags contain HTML entities.

1
2
3


<meta property='og:description' content='I work at AWS, and predictably we use a lot of AWS cloud services. In many cases, when an engineer looks for a computer platform, they&rsquo;ll often go directly to AWS Lambda because &ldquo;it&rsquo;s Serverless&rdquo; with the justification that it&rsquo;s simple and the best option no matter what and not want to explore alternatives. The FaaS (Functions as a Service) compute style is great for a certain category of system problems&ndash;ones in which you don&rsquo;t need strict control over how it executes. AWS Lambda only exposes limited controls and depending on what your workload is like, you could run into unexpected scaling and failure modes. There are alternative compute environments that avoid those limitations that you should know about. '>

<meta property='article:published_time' content='2024-09-29T00:00:00&#43;00:00'/><meta property='article:modified_time' content='2024-09-29T00:00:00&#43;00:00'/>

My Hugo template had a file that looked like this:

1
2
3
4
5
6
7


{{- $title := partialCached "data/title" . .RelPermalink -}}
{{- $description := partialCached "data/description" . .RelPermalink -}}

<meta property='og:title' content='{{ $title }}'>
<meta property='og:description' content='{{ $description }}'>
<meta property='og:url' content='{{ .Permalink }}'>
<meta property='og:site_name' content='{{ .Site.Title }}'>

If we look at Hugo’s repo in the opengraph.html template, they use plainify to remove HTML tags, then htmlUnescape to remove HTML entities and encoding from the strings.

1
2
3


{{- with or .Description .Summary site.Params.description | plainify | htmlUnescape }}
 <meta property="og:description" content="{{ trim . "\n\r\t " }}">
{{- end }}

Solution

Let’s do a similar fix.

1
2
3
4
5
6
7


{{- $description := partialCached "data/description" . .RelPermalink -}}
{{- $description := $description | plainify | htmlUnescape }}
{{- $description := trim $description "\n" -}}
{{- $description := $description | safeHTMLAttr -}}

<meta property='og:title' content='{{ htmlUnescape .Title | safeHTMLAttr }}'>
<meta property='og:description' content='{{ $description }}'>

Now we get the following HTML:

1
2


<meta property='og:description' content='I work at AWS, and predictably we use a lot of AWS cloud services. In many cases, when an engineer looks for a computer platform, they’ll often go directly to AWS Lambda because “it’s Serverless” with the justification that it’s simple and the best option no matter what and not want to explore alternatives.
The FaaS (Functions as a Service) compute style is great for a certain category of system problems–ones in which you don’t need strict control over how it executes. AWS Lambda only exposes limited controls and depending on what your workload is like, you could run into unexpected scaling and failure modes. There are alternative compute environments that avoid those limitations that you should know about.'>

And now my OpenGraph tags are generated correctly.

ActivityPub JSON

Investigation

My ~~secret~~ upcoming project is adding ActivityPub support to this blog. It’s not finished yet and a separate post will be made, but here we can see a similar encoding problem. All the content was being HTML escaped which meant that links were not clickable and were directly visible in the post.

This is exactly the same problem as before. Say I’ve got a file: layouts/posts/single.post_json.json that generates the ActivityPub JSON for a single post with the following subset:

1
2
3
4
5
6
7


{
 "id": "{{ .Permalink }}",
 "type": "Article",
 "content1": {{ printf "%s" .Summary | jsonify }},
 "content2": {{ .Content | htmlUnescape | jsonify }},
 "content3": "{{.Summary}}",
}

Looking at these two encoding types, which one do you think is correct? Let’s get a complex test post that contains quotation marks:

1
2
3
4
5
6


---
title: test \"test2"
author: "author\""
---

test "test"<br> [test](https://example.com). test"

Running hugo gives me the following output file:

1
2
3
4
5
6


{
 "content1": "\u003cp\u003ehere \u0026ldquo;is\u0026rdquo; a summary\u003cbr\u003e \u003ca class=\"link\" href=\"https://example.com\" target=\"_blank\" rel=\"noopener\"\n \u003etest\u003c/a\u003e. test\u0026quot;\u003c/p\u003e",
 "content2": "\u003cp\u003ehere “is” a summary\u003cbr\u003e \u003ca class=\"link\" href=\"https://example.com\" target=\"_blank\" rel=\"noopener\"\n \u003etest\u003c/a\u003e. test\"\u003c/p\u003e\n",
 "content3": "<p>here &ldquo;is&rdquo; a summary<br> <a class="link" href="https://example.com" target="_blank" rel="noopener"
 >test</a>. test&quot;</p>",
}

Solution

Looking at content3, it incorrectly serializes any quotation marks and generates corrupted JSON syntax, so that’s out. Looking at content1, we see examples of HTML entities, like \u0026ldquo; . That’s why we’re seeing the HTML tags written raw in Mastodon, instead of as a clickable link as we expected. content2 does not include any HTML entities, thus is the correct format.

Those \u003c Unicode encodings are extraneous, we can use jsonify’s options to disable them. Here’s the final:

1
2
3
4
5


{
 ...
 "content": {{ .Content | htmlUnescape | jsonify (dict "noHTMLEscape" true) }},
 ...
}

This pattern of string | htmlUnescape | jsonify (dict "noHTMLEscape" true) should only be used when you intentionally want to emit HTML tags into a JSON field value. Don’t allow user defined content to enter these values without sanitizing it, but if you’re using Hugo, it’s probably all static defined by you.

URL Encoding inside RSS/XML

My next problem was to embed an image into an RSS feed. At first I did:

1

<img referrerpolicy="no-referrer-when-downgrade" src="https://www.technowizardry.net/rss_view?action_name={{ .Title }}&amp;url={{ .Permalink }}" style="border:0" alt="" />

That incorrectly generates the following HTML:

1

&lt;img referrerpolicy="no-referrer-when-downgrade" src="https://www.technowizardry.net/rss_view?action_name=Fixing common Hugo encoding problems&amp;url=https://www.technowizardry.net/2025/03/fixing-common-hugo-encoding-problems" style="border:0" alt="" /&gt;

Multiple issues here:

The HTML tags use <> instead of <>
The title and the URL is not URL encoded, so the request doesn’t get serialized correctly.

Solution

To serialize the HTML tags, we change from < to {{ "<" | html }} and to generate the URL, we use {{ .Permalink | urlquery }}. The urlquery method URL-encodes each variable to fit within the URL parameters.

1

{{ "<" | html }}img referrerpolicy="no-referrer-when-downgrade" src="https://www.technowizardry.net/rss_view?url={{ .Permalink | urlquery }}%3Fmtm_campaign%3Drss&amp;action_name={{ .Title | urlquery }}" style="border:0" alt="" {{ "/>" | html}}

Summary

By default, Hugo tries to escape every single string to avoid emitting HTML elements where they don’t belong. This is a good security mechanism, but sometimes you need to generate non-HTML files, like JSON, or sometimes you want the HTML elements to emitted in an output artifact without them being escaped.

If you want HTML elements to be removed and are emitting into an HTML element, do this:

1

<meta property="og:description" content="{{ .Summary | plainify | htmlUnescape }}">

If you’re emitting into a JSON file, do this:

1

"field": {{ .Content | htmlUnescape | jsonify (dict "noHTMLEscape" true) }}

I hope this helps. Stay tuned for further posts on ActivityPub

Email spam filtering with Rspamd on K8s

Mon, 10 Mar 2025 21:07:00 +0000

I’ve been running my own mail server for well over ten years now. It’s pretty old, so it’s hard to make changes to it, but it’s running in Kubernetes. I was using a mixture of Postfix, OpenDKIM, OpenDMARC, and Amavis for spam filtering with SpamAssasin, but it wasn’t very good at catching spam. Instead, its time move to rspamd. It’s much newer and encapsulates DKIM, DMARC, DNS based blacklisting, bayesian filtering, etc. all in one single tool.

Here’s my notes on migrating, what it took to get it going and some tweaks I made.

Setup Redis/Valkey

Rspamd uses Redis to store runtime stats and config. Deploy it using Helm, or manually. I’m going to assume it’s deployed in the mail namespace with a service named called rspamd-redis.

Rspamd Configuration

I started with a basic configuration defined below. Starting with this meant I could skip the quick start guide that mandated that I run the rspamadm configwizard command, though you might have some slightly different needs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


apiVersion: v1
data:
 classifier-bayes.conf: |
 servers = "rspamd-redis.mail.svc.cluster.local.";
 backend = "redis";
 redis.conf: |
 read_servers = "rspamd-redis.mail.svc.cluster.local.";
 write_servers = "rspamd-redis.mail.svc.cluster.local.";
 worker-normal.inc: |
 bind_socket = "0.0.0.0:11333";
 worker-proxy.inc: |
 milter = yes; # Enable milter mode
 timeout = 120s; # Needed for Milter usually
 upstream "local" {
 default = yes; # Self-scan upstreams are always default
 self_scan = yes; # Enable self-scan
 }
kind: ConfigMap
metadata:
 name: rspamd-config
 namespace: mail

Then deployed rspamd like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: rspamd
 namespace: mail
spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 256Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: rspamd
 namespace: mail
spec:
 replicas: 1
 selector:
 matchLabels:
 app: rspamd
 component: main
 template:
 metadata:
 labels:
 app: rspamd
 component: main
 spec:
 containers:
 - image: rspamd/rspamd
 imagePullPolicy: IfNotPresent
 name: rspamd
 volumeMounts:
 - mountPath: /etc/rspamd/local.d
 name: config
 - mountPath: /var/lib/rspamd
 name: data
 subPath: data
 volumes:
 - configMap:
 defaultMode: 420
 name: rspamd-config
 name: config
 - name: data
 persistentVolumeClaim:
 claimName: rspamd

Then in postfix, I remove all the existing filters and use only rspamd:

1
2
3
4
5


- non_smtpd_milters = inet:localhost:9999, inet:localhost:9998,
- content_filter = amavisfeed:[127.0.0.1]:10024

+ smtpd_milters = inet:rspamd.mail.svc.cluster.local:11332
+ non_smtpd_milters = inet:rspamd.mail.svc.cluster.local:11332

DNS Blocklisting

Upon starting rspamd, I get a bunch of warnings like this:

1

rspamd_monitored_dns_cb: DNS reply returned 'no error' for zen.spamhaus.org while 'no records with this name' was expected when querying for '1.0.0.127.zen.spamhaus.org'(likely DNS spoofing or BL internal issues)

It filters email, but still lets through a lot of obvious spam. This happens because rspamd queries several DNS-based block-list providers, like Spamhaus, to identify which IP addresses commonly send spam and reject them. But providers, like Spamhaus, have a free-tier and paid tiers and chose to block public DNS servers because public DNS servers anonymize the source for queries and Spamhaus is unable to limit traffic. I’m a very low-volume mail receiver, so the free tier is fine.

My server was just using the default DNS server provided by my dedicated server provider, OVH, which they considered to be a public resolver. Thus every query was being rejected.

To fix this, I’m going to have to change my resolver to not use a public resolver and instead use a resolver that does not use a public resolver. Instead, I’ll use a resolving DNS server, one that sends queries directly to Spamhaus, i.e. a recursive resolving DNS server. I just needed a simple containerized server. CoreDNS, which I normally use, did not support recursive DNS queries without an external plugin. Instead, I came across unbound which natively did.

Configuration

The following is the configuration I used. The server is configured to forward queries for .cluster.local. to the Kubernetes internal kube-dns instance. This is required because rspamd needs to connect to other Kubernetes services, such as Redis.

Additionally, we must allow all DNS blocklist domains to return internal IP addresses via the private-domain configuration option. This is critical because DNS-based block lists commonly use A DNS queries that return IP addresses to signal what action a mail server should take. For example, Spamhaus returns 127.0.0.2 when a request is on a specific filter list (other responses.) Without this configuration, unbound would discard all responses.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


apiVersion: v1
data:
 unbound.conf: |
 server:
 chroot: ""

 # Skip IPv6 since my cluster is not dual-stack yet
 do-ip6: no

 private-domain: cluster.local.
 # Add all DNS blocklists below
 private-domain: dnsbl.manitu.net.
 private-domain: surbl.org.
 private-domain: spamhaus.org.
 private-domain: spameatingmonkey.net.

 # Forwards all queries for internal cluster queries to kube-dns
 forward-zone:
 name: "cluster.local."
 forward-addr: 10.43.0.10@53
kind: ConfigMap
metadata:
 name: rspamd-dns-resolver
 namespace: mail

Switch to unbound

Next, we need to start the DNS server. I added it as a side-car container with the existing Rspamd service and configured the pod’s dnsConfig to unbound.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


apiVersion: apps/v1
kind: Deployment
metadata:
 name: rspamd
 namespace: mail
spec:
 template:
 spec:
 containers:
 - name: rspamd
 # ... elided
 - image: mvance/unbound:latest
 imagePullPolicy: IfNotPresent
 name: dns
 resources:
 limits:
 memory: 128Mi
 requests:
 cpu: 8m
 memory: 64Mi
 volumeMounts:
 - mountPath: /opt/unbound/etc/unbound/unbound.conf
 name: dns-config
 subPath: unbound.conf
 dnsConfig:
 nameservers:
 - 127.0.0.1
 options:
 - name: ndots
 value: '1'
 dnsPolicy: None
 volumes:
 # ...
 - configMap:
 defaultMode: 420
 name: rspamd-dns-resolver
 name: dns-config

After restarting Rspamd, it started using the DNS block lists and was able to flag even more obvious spam.

Conclusion

Migrating to rspamd from my previous solution was a great idea. It consolidated and reduced the services I had to run and it actually helped cut down on the spam that made it through with better rules. The UI was very easy to investigate what emails were being marked and why.

My previous solution based on Amavis and SpamAssassin was a black-box and it was difficult to understand what’s going on without digging through logs.

Lighting up the holidays with computers

Tue, 14 Jan 2025 00:00:00 +0000

This Christmas season, I decided I wanted to play with programmable light strings and see if I could create an interesting light show on the front of my house. I stumbled across xlights, an open source light show sequencing program and got to work.

The Hardware

My light set-up needed a central light controller that would store the light sequences, send the light data to the strings of lights. There were a few different vendors, but the Xlights communities generally recommended Kulp or Falcon. I selected a Kulp K8-B controller because Kulp controllers were standard BeagleBoard computers that ran standard Linux which gave me some flexibility. I ordered the controller kit through the build a controller kit tool on WiredWatts.com and some other lights/cables through other vendors.

I ended up with a basic selection with:

Kulp K8-B - The light controller
A waterproof enclosure
A 350 watt power supply
Two icicles pixel strings
Two flood lights (I plan to illuminate some trees)
Two strings of more efficient, string pixels

The kit came not assembled and contained no instructions on how to put it together. I mostly just put it together with trial and error.

Electrical Power

Power management is important, but I didn’t realize how important it was. I got the lights up on the house and started testing some lights. It was daytime and the lights defaulted to 100% brightness. The lights flashed white, then a few seconds later they didn’t turn on. Weird. I thought it was a software issue until I thought to look at the controller which contains a number of fuses and some status LEDs showing if they are working. And low and behold, I blew a fuse.

Let’s break this down. I’ve got one outdoor outlet that’s rated for 15 amps (1600 watts at 110v). That feeds into a 350 watt power supply at 12v DC (29 amps) that powers the Kulp controller. The controller has eight separate light ports that each have a 5amp fuse. I then connected two strings of lights that each peaked at 5 amps consumption, thus 10 amps total on a 5 amp fuse for 100% brightness and full white. Oops.

At night, the LEDs are so bright that they can easily be run at only 10-15% brightness which helps to cut the power utilization.

It would be really helpful if I were able to measure the current consumption across the entire power supply and on individual ports to identify when I’m getting close to the limits.

Laying the lights out in xLights

For example, I have icicles that wrap around my front porch. Like this:

If I were to create an effect that goes from one side to the other, xLights would have to figure out how to apply the effect to the pixels. If I apply a simple slide effect from one side to another, it’ll look like the following. Notice how the sides of the icicles seem to fill immediately, then the center fills smoothly.

Let’s create a sample effect: Fill. Click Direction: Left.

Then click the green, right-pointing arrow for Position. Then pick a linear ramp in the bottom row of grey buttons and click okay.

Then drag that onto the sequencer

After clicking the render button , I can then see what the effect looks like in the left preview window.

Fixing Rendering issues

xLights doesn’t know what I mean and what I want, so I need to tell it how it should map this shape. Luckily it provides a feature to fix this that is poorly documented, but commonly used to fix this.

A Shadow Model is a copy of a model that you can reposition in different ways to get xLights to render effects the way you expect it to. I created two icicles models to represent the sides of the patio, then extended them out laterally like they’re sticking out instead of wrapping around the sides. The width, height, and number of light drops should match. From a 2D view looking straight on, it looks like this:

Then each of those get marked as a Shadow Model for the left/right icicles.

Then I create a model group for all my patio icicles that contains the shadow group instead of the actual lights.

If I re-apply my effect, I get this:

The Light Show (Sequencing)

Now the ~~hard~~ fun part. Actually coming up with some light shows. I’m not good at this at all. I initially wanted to do something to some songs, but that requires some kind of audio transmitter, like an FM transmitter, speaker, or even online streaming. Also, figuring out what effects should go with different parts of the song, was not easy for me being rhythmically challenged.

Instead, I started simple. A few simple, aesthetically pleasing effects on the lights. For example, my first sequence included just 30 seconds of a meteor effect, plus some subtle lighting of trees in the front yard. One of the Japanese maple trees has beautiful red leaves (until they dropped in the winter), so adding a bit of red light, caused the leaves to pop in color.

A Video

Conclusion

I had fun playing with Xlights and getting a basic light show set-up. The sequencing is complicated and it took me some time to figure out how to build the exact effect that I wanted. Next year, I’m going to expand it and add more lights.

More infrastructure doesn't fix using the wrong infrastructure

Sun, 29 Sep 2024 00:00:00 +0000

I work at AWS, and predictably we use a lot of AWS cloud services. In many cases, when an engineer looks for a computer platform, they’ll often go directly to AWS Lambda because “it’s Serverless” with the justification that it’s simple and the best option no matter what and not want to explore alternatives.

The FaaS (Functions as a Service) compute style is great for a certain category of system problems–ones in which you don’t need strict control over how it executes. AWS Lambda only exposes limited controls and depending on what your workload is like, you could run into unexpected scaling and failure modes. There are alternative compute environments that avoid those limitations that you should know about.

Sure, there is a cost to understanding a new service like having to learn a new query language. What I realize is that not all engineers assign the same weight to that knowledge. There’s different uncertainty in the new and different.

However, in this post I make the claim that FaaS platforms aren’t the best solution for everything and you should recognize situations where they are good and are not good.

The Example - Data Processing Lambda

In this example, you have a situation where you need to take a file from S3, iterate over it, and do some useful work on it. Maybe you save it to another file, maybe push it into a service. Another developer wants to run a Lambda function that does this. It could work right? Lambda can fetch files, process them, and write them back somewhere else. What’s wrong with that?

Let’s take a look at an example implementation as a Lambda function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


public class Program implements RequestHandler<Request, Response> {
 public Response handleRequest(Request event, Context context)
 {
 S3Client s3 = S3Client.create();
 String fileContent = readS3File(s3);
 String output = multiplyValue(fileContent);
 writeOutputToS3(s3, "s3://outputbucket/foobar", output);
 return null;
 }

 private static String readS3File(S3Client s3, String filePath) throws IOException {
 ResponseBytes bytes;
 GetObjectRequest request = GetObjectRequest.builder()
 .bucket("awsexamplebucket-streaming-demo2")
 .key("inputs/productsStatic.csv")
 .build();
 try {
 bytes = s3.getObjectAsBytes(request);
 } catch (IOException e) {
 // TODO
 }
 return new String(bytes.content().array());
 }

 private static int performBusinessLogic(int value) {
 // Here's the actual business logic right here
 return value * 10;
 }

 private static String multiplyValue(String fileContent) {
 Scanner scanner = new Scanner(fileContent);
 StringBuilder output = new StringBuilder();
 while (scanner.hasNextLine()) {
 String line = scanner.nextLine();
 String[] values = line.split(",");
 // TODO: Handle errors with the input file
 String multipliedValue = String.valueOf(Integer.parseInt(performBusinessLogic(values[1])));
 output.append(multipliedValue).append("\n");
 }
 return output.toString();
 }

 private static void writeOutputToS3(S3Client s3, String filePath, String content) {
 try (FileWriter writer = new FileWriter("output.csv")) {
 writer.write(content);
 s3.putObject(PutObjectRequest.builder()
 .bucket("outputbucket")
 .key("foobar/output.csv")
 .build(),
 SdkBytes.fromInputStream(new FileInputStream("output.csv")));
 } catch (IOException e) {
 // TODO: Handle errors and retries
 }
 }
}

Is that bad? Should you push back? You might think it’s pretty simple to call s3.getObject, load it, parse it, etc. Why bother with something else? Well, what do you do if the data set starts getting larger and you run into the 15 minute Lambda time limit?

Just split it into two jobs and have one Lambda function perform part of the work, then the second Lambda function perform the rest. (Think that’s crazy? This actually happened in one project I worked with.) Now you have an orchestration problem. What happens if one function succeeds, but the second fails? You have to setup monitoring rules for both and try to handle retries.

Or maybe you try run multiple threads to process in parallel. Now, you’ve got another type of orchestration: batching up work to each thread, waiting for responses, making sure you have your threads scaled to the number of Lambda cores.

Can you solve these problems? Yes. Should you, probably not. Why? They are not solving the actual business problem. The system is there to provide an answer, anything else is just getting in the way. Using the wrong tool in the toolbox makes your life harder. Picking the right tool, i.e. the right framework, makes it easier.

Trying another compute environment

What about AWS Batch or AWS Fargate? These are two hosted compute environments (AWS still owns the underlying OS), but you own the main method. Thus you have more control over the life-cycle of your process. It does away with the 15 minute limit and gives you some controls on what kind of host you’re running on.

That means you don’t have to worry about 15 minute timeouts. But you still have other edge cases: the call to S3 could fail, the file format changes unexpectedly, etc. Your code has to handle that, if not at first, then eventually at 3am when somebody gets paged when it breaks or changes.

The ETL Job

Looking at it from a different perspective, this could be an ETL (Extract, Transform, Load) job. An ETL job is a high-level concept that takes an input file, transforms it in some way, maybe joins it with another file, then writes it to another file. Exactly what the code above does.

This is exactly the kind of thing that ETL services are great at, such as AWS Glue, or AWS EMR, this would look like the following (some lines removed (see here for full example.) AWS Glue uses Spark and you can either use conventional SQL or define your own Spark Scala functions. AWS EMR can run several different big data compute frameworks, but for this post we’re going to use Spark because I’m familiar with it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


object Program {
 def main(sysArgs: Array[String]) {
 // Yes this is boiler-plate code not relevant for your business logic
 // But it's smaller and limited to just the start
 val spark: SparkContext = new SparkContext()
 val glueContext: GlueContext = new GlueContext(spark)
 val sparkSession: SparkSession = glueContext.getSparkSession
 import sparkSession.implicits._

 val input = sparkSession.read
 .format("csv")
 .option("header", "true")
 .load("s3://awsexamplebucket-streaming-demo2/inputs/productsStatic.csv")

 input.select(($"value" * 10) as "value")
 .write
 .save("s3://outputbucket/foobar")
 }
}

There’s some boiler-plate in here just setting up the Spark system, but the actual business logic is simple. It’s a declarative model (you describe what you want done) and it makes it happen instead of the previous example being imperative (you describe how it happens.)

Is it really cheaper to go learn Spark if you weren’t already familiar? If it was just this small snippet of code, probably not. But systems rarely stay the same as what they were originally envisioned. The growing problems may or may not come up in a prototype that you do just to try it out.

When you’re writing everything from scratch, you have to deal with all the minutia of the problem. Did you do a retry? Should you run it in multiple threads for performance? How do you parse the CSV file? It gets really easy to just layer more and more on it because the alternative is unknown because either you don’t know it exists, or you don’t know how hard it’ll be to use.

Soon enough, you’re spending your time fixing issues in the system that aren’t the important parts (the business logic.) And personally, as a developer, I don’t enjoy running around fixing those kinds of issues.

But Spark SQL is weird and hard

A common response I get about frameworks like Spark is that Scala is confusing, or Spark SQL influences too much of your code.

Yes, the Spark SQL functions are different than native Java, Python, or Scala and they can get a bit complicated modeling everything inside of Spark SQL. The following is what the FizzBuzz problem looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


val df = spark.createDF(Seq(1 to 100)) // TODO: Validate this

val mod3 = $"value".mod(lit(3)) == 0
val mod5 = $"value".mod(lit(5)) == 0

df.withColumn("fizzbuzz",
 when(
 mod3.and(mod5),
 lit("fizzbuzz")
 )
 .when(mod3, lit("fizz"))
 .when(mod5, lit("buzz"))
 .otherwise($"value".cast("string"))
 )

Going 100% into that style of coding changes from one problem to another. You went from imperative code that doesn’t handle edge cases, to one where the execution framework heavily dictates your coding style. For some engineers, this will also be confusing.

There are some solutions. You can develop a hybrid code style where you can pull out business logic into conventional Scala, Python, or Java because Spark Scala has full interop with JVM languages and PySpark has interop with Python.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


val df = spark.read.parquet("s3://blah")

val fizzbuzz = (value) => {
 if ((value % 3) & (value % 5)) {
 "fizzbuzz"
 } else if (value % 3) {
 "fizz"
 } else if (value % 5) {
 "buzz"
 } else {
 value.toString
 }
})

df.select(udf[String, Int](fizzbuzz)($"value"))

If you have a large block of business logic and you only want Spark to handle input/output, task distribution, retries, etc. you can just jump out to native Java/Scala/Python code to perform the business logic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


val df = spark.read.parquet("s3://blah")

def fizzBuzzPartition(rows: Iterator[Row]) => {
 rows.map(row => {
 val value = row.getInt(0)
 // Imagine fizzbuzz() is calling out to existing business logic
 return Row(fizzbuzz(value))
 })
}

val out = df.mapPartitions(fizzBuzzPartition)

Should you always use the “best” technology?

Nah. Even if there appears to be a technology that seems to abstract away all your problems, you can’t ignore the other costs like having to learn a new language, new framework, new failure modes, and operational monitoring requirements. Your teammates have to learn something new and that adds a lot of friction. Some engineers don’t want or have the energy to learn something new.

You have to weigh the benefits with the costs. If it’s a component you’re not going to tough very frequently, err on the side of technologies that are extremely simple and easy for others to understand. Pick ones that are consistent with others.

If it is a key component that you’re going to continue to develop on, then it’s worth spending some extra time to understand these other technologies.

Conclusion

Picking frameworks and technologies is like picking the right tool in the toolbox. Sure you can probably make squeaky door hinge stop with some WD-40, but you really want some lubricant. The right framework is important because software in production has a lot of hidden edge cases that have to be handled.

Don’t always pick the fancy technology. Weigh the benefits and the costs. Using the technology you already know is a big benefit and something new should be considered a cost. But don’t be overly confident in your ability to identify and handle all edge cases.

If you feel like you’re bending a system or framework to get it to work, take a step back and ask yourself if you’re using the right solution.

Fixing an IP conflict with Docker and Delta in-flight wi-fi

Fri, 13 Sep 2024 00:00:00 +0000

Today, I took a flight and tried to use the in-flight Wi-Fi, but I was unable to login to the the network. Nothing loaded or opened. I poked around in ip route and found two different routes that conflicted created by the Docker daemon. Looking at the following route, there’s two routes: 172.19.0.0/23 and 172.19.0.0/16. These correspond to: 172.19.0.0 - 172.19.1.255 and 172.19.0.0 - 172.19.255.255.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


ip route

default via 172.19.0.1 dev wlp1s0 proto dhcp src 172.19.1.22 metric 600
172.18.0.0/16 dev br-71fb429d0dc2 proto kernel scope link src 172.18.0.1 linkdown

vvv
172.19.0.0/23 dev wlp1s0 proto kernel scope link src 172.19.1.22 metric 600
172.19.0.0/16 dev br-908cdaa5e0f4 proto kernel scope link src 172.19.0.1 linkdown
^^^

172.20.0.0/16 dev br-5432e3728889 proto kernel scope link src 172.20.0.1 linkdown
172.31.0.0/24 dev docker0 proto kernel scope link src 172.31.0.1 linkdown

And here’s the problematic networks:

1
2
3
4
5
6
7
8
9


docker network ls

NETWORK ID NAME DRIVER SCOPE
908cdaa5e0f4 testwebsite_default bridge local
b112ed129840 bridge bridge local
cd3a9ccce4c5 host host local
2e34dc80f263 none null local
5432e3728889 quickstart bridge local
71fb429d0dc2 technowizardrynet_default bridge local

First step is to delete these networks docker network rm 908cdaa5e0f4 5432e3728889.

Then configure Docker to use a new range by adding the following to /etc/docker/daemon.json:

1
2
3
4
5


{
 "default-address-pools": [
 { "base":"172.31.0.0/16", "size":24 }
 ]
}

And restart docker: sudo systemctl restart docker. After that, you should be able to use Docker and the in-flight Wi-Fi.

Screen sharing on Wayland Ubuntu 24.04

Tue, 10 Sep 2024 00:00:00 +0000

I tried screen sharing in a video call on my Ubuntu 24.04 computer running the Snap Firefox install, but I could never get it to prompt to share a screen, thus it wouldn’t work. This post shows how I fixed that.

Investigation

My initial web searches came across this Stack Exchange question that recommended that users switch from Wayland to X11. While this did fix the problem, I’m not a fan of just turning off new technologies to get things to work, so I continued investigating.

GNOME and desktop environment integrations usually appear in D-Bus and the awesome packet capturing tool, Wireshark, has the ability to sniff D-Bus messages.

I started capturing packets, then went back to Firefox and tried to share my screen. From there, I was able to see a few interesting packets:

And an error message stating No such interface "org.freedesktop.portal.ScreenCast". Searching for that lead me to this comment.

D-Spy is also a useful tool for investigating these issues too. With it, I’m able to see that Screencast is enabled.

To enable it, run: sudo apt install d-spy

The Fix

Easy enough: sudo apt install xdg-desktop-portal xdg-desktop-portal-gnome

I installed those packages, then rebooted. Voilà! Firefox is now able to share my screen.

Adopting NixOS for my RKE1 Kubernetes nodes

Sat, 07 Sep 2024 00:00:00 +0000

For those not aware, Nix is an interesting new application (Nix) and operating System (NixOS) that provides a declarative environment definition and atomic operating system. Declarative means that instead of running apt-get install docker, you write down everything you want and it installs everything and removes everything you don’t want. You can use the same language to manage packages, users, firewall, networking, etc. This is useful because now you can revision control your OS state in Git and have exact replicas across multiple hosts.

My friend, dade, and I have been diving into Nix and NixOS. He got it working on his laptop, I’m trying to get it to be the OS for my four dedicated servers all running Kubernetes. In this post, I’ll walk through the main issues I encountered and how I got a single node running in an existing RKE1 cluster.

I’m not going to go all the way to use Nix to configure everything including my Kubernetes configuration. I know that’s possible, but I already have a Kubernetes cluster deployed using RKE1 that I’m not ready to break yet since it hosts this blog and other services. Maybe in a future iteration I will.

Booting

First, I needed to make the disk as a bootable EFI image:

1
2
3
4
5
6
7


{ config, lib, pkgs, ... }:
{
 # Use the systemd-boot EFI boot loader.
 boot.loader.systemd-boot.enable = true;

 boot.loader.efi.canTouchEfiVariables = true;
}

Disk Partitioning

The NixOS minimal live CD does not give a partitioning tool other than fdisk. You’re responsible for figuring out partition types yourself. Luckily NixOS has disko which uses the exact same Nix files to declaratively define partitions and I can steal the config from somebody else who figured it out. Nice.

I had two drives, one SSD, one on a spinning hard drive and followed the disko guide. My configuration looked like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57


{ config, lib, pkgs, disko, ... }:
{
 disko.devices = {
 disk = {
 ssd = {
 # When using disko-install, we will overwrite this value from the commandline
 device = "/dev/disk/by-id/5dbd3913-bc15-4e14-9902-de9c1703eacd";
 type = "disk";
 content = {
 type = "gpt";
 partitions = {
 MBR = {
 type = "EF02"; # for grub MBR
 size = "1M";
 priority = 1; # Needs to be first partition
 };
 ESP = {
 type = "EF00";
 size = "500M";
 content = {
 type = "filesystem";
 format = "vfat";
 mountpoint = "/boot";
 };
 };
 root = {
 size = "100%";
 content = {
 type = "filesystem";
 format = "btrfs";
 mountpoint = "/";
 };
 };
 };
 };
 };
 hdd = {
 # When using disko-install, we will overwrite this value from the commandline
 device = "/dev/disk/by-id/eae13bd2-c505-46b2-9066-5aa1028f10d7";
 type = "disk";
 content = {
 type = "gpt";
 partitions = {
 root = {
 size = "100%";
 content = {
 type = "filesystem";
 format = "btrfs";
 mountpoint = "/mnt/hdd";
 };
 };
 };
 };
 };
 };
 };
}

Writing to the disk was easy (make sure sdb and sda are pointing to the right disks)

1

nix --extra-experimental-features flakes --extra-experimental-features nix-command run 'github:nix-community/disko#disko-install' -- --flake '/tmp/config/etc/nixos#srv5' --write-efi-boot-entries --disk ssd /dev/sdb --disk hdd /dev/sda

Making Vim sane

Nix’s default vimrc configuration is weird for me, but it’s easy to fix this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


environment.systemPackages = with pkgs; [
 ((vim_configurable.override { }).customize {
 name = "vim";
 vimrcConfig.customRC = ''
 set mouse=""
 set backspace=indent,eol,start
 syntax on
 '';
 })
];

Moving on to K8s

I’m currently using RKE1 which involves running ./rke up to provision a new node, not NixOS because my K8s is a few years old now. To add a new node it looks like:

1
2
3
4
5
6
7
8


 # cluster.yml
 nodes:
+ - address: 1.2.3.4
+ user: adam
+ role:
+ - controlplane
+ - etcd
+ - worker

Then run: (--ignore-docker-version is needed because NixOS uses Docker 27.x, but RKE1 only supports up to 26.x)

1

./rke_linux-amd64 up --ignore-docker-version

And waiting for it to provision.

Resource requests (The dumb issue)

The node joined the cluster, but my next problem is no pods could start because the networking was down:

The kubelet logs (docker logs kubelet) showed:

1
2
3


kubelet.go:2862] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"

0/4 nodes are available: 1 Insufficient cpu, 3 node is filtered out by the prefilter result. preemption: 0/4 nodes are available: 1 Insufficient cpu, 3 Preemption is not helpful for scheduling..

I was stumped. How come there wasn’t enough CPU? Was it because Calico couldn’t spin up, so kubelet was reporting unhealthy because there was no CNI, thus no CPU? A circular dependency?

Nah, let’s get the dumb mistakes out of the way: turns out, I only provisioned one CPU to this test VM because it was just for experimenting, not production use.

And I had reserved 1 core for the host OS:

1
2
3
4
5


# rke cluster.yaml
services:
 kubelet:
 extra_args:
 kube-reserved: cpu=1000m,memory=512Mi

So there was nothing available for the Kubernetes pods itself. Whoops. I gave it more CPUs.

Inter-node traffic

Next problem is diagnosing why the pods weren’t able to communicate with the rest of the cluster. I’m currently using Calico Canal for networking which uses Flannel to create an encapsulated overlay network between the different nodes in the cluster. They weren’t able to communicate and I needed to figure out why, so I installed tcpdump, a packet capturing program. I added the following, then ran nixos-rebuild switch:

1
2
3


environment.systemPackages = with pkgs; [
 pkgs.tcpdump
];

On a different node, I pinged a pod running on my test node to identify which ports were needed:

1
2
3


ping 10.42.4.72

PING 10.42.4.72 (10.42.4.72) 56(84) bytes of data.

Gotcha:

1
2
3
4


tcpdump -f 'host 51.81.64.31 or dst 149.56.22.10'

05:04:55.054420 ens33 Out IP srv8.7946 > srv6.technowizardry.net.7946: UDP, length 96
05:04:55.068243 ens33 In IP srv6.technowizardry.net.7946 > srv8.7946: UDP, length 49

Easy enough in Nix to open up the firewall. Ideally, I’d lock this down to just the other nodes in my cluster, but I didn’t know how to do that in NixOS yet.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{ config, lib, pkgs, ... }:
{
 networking.firewall.allowedTCPPorts = [
 7946 # flannel
 8285 # flannel
 8472 # flannel

 # Kubernetes
 2379 # etcd-client
 2380 # etcd-cluster
 6443 # kube-apiserver

 # Prometheus metrics
 10250
 10254
 ];
 networking.firewall.allowedUDPPorts = [
 7946 # flannel
 8472 # flannel
 ];
}

Another NixOS rebuild and the pings are flowing between pods.

(it’s always) DNS

All my servers (via a DaemonSet) run PowerDNS as an authoritative DNS server for my various domain names. However, on my new node, it wasn’t able to start up:

1
2
3


Guardian is launching an instance
Unable to bind UDP socket to '0.0.0.0:53': Address already in use
Fatal error: Unable to bind to UDP socket

It binds to port 53/tcp and 53/udp on 0.0.0.0, however systemd-resolved is already running on that port on 127.0.0.53 and 127.0.0.54. Even though it’s listening on loopback, PowerDNS will conflict because it’s trying to run on 0.0.0.0.

1
2
3
4
5
6
7
8


lsof -iUDP -P -n
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-r 692 systemd-resolve 11u IPv4 14287 0t0 UDP *:5355
systemd-r 692 systemd-resolve 13u IPv6 14290 0t0 UDP *:5355
systemd-r 692 systemd-resolve 15u IPv4 14292 0t0 UDP *:5353
systemd-r 692 systemd-resolve 16u IPv6 14293 0t0 UDP *:5353
systemd-r 692 systemd-resolve 20u IPv4 14297 0t0 UDP 127.0.0.53:53
systemd-r 692 systemd-resolve 22u IPv4 14299 0t0 UDP 127.0.0.54:53

To fix this, I disable the local resolver and point my server to another resolver:

1
2
3
4
5
6


{ config, lib, pkgs, ... }:
{
 networking.nameservers = [ "1.1.1.1" ];

 services.resolved.enable = false;
}

Longhorn

I use Longhorn as my Kubernetes CSI (storage provider) because it supports mirroring across multiple hosts and automatic backups. Under the hood, it uses NFS to mount a volume from the Longhorn storage pod and the host running the application pod. To do this, it requires a few packages to be installed on the host itself (nfs-utils) and the iscsid system daemon.

When I tried to start Longhorn, I got the following error messages because the nfs utils weren’t installed:

1
2
3
4
5
6


MountVolume.MountDevice failed for volume "pvc-8416dbbf-89cf-45c0-bdc8-8a2b55a4e58a" :
rpc error: code = Internal desc = mount failed: exit status 32 Mounting command:
/usr/local/sbin/nsmounter Mounting arguments:
mount -t nfs -o vers=4.1,noresvport,timeo=600,retrans=5,softerr 10.43.149.52:/pvc-8416dbbf-89cf-45c0-bdc8-8a2b55a4e58a /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/6d6bbeebe575c0ca8e25eb35c0248aca3e81a606bc647be0c2d9901b6b4bd9b3/globalmount Output:
mount: /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/6d6bbeebe575c0ca8e25eb35c0248aca3e81a606bc647be0c2d9901b6b4bd9b3/globalmount:
bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program. dmesg(1) may have more information after failed mount system call.

As per the Longhorn docs, we need to install a few packages:

1

environment.systemPackages = [ pkgs.nfs-utils pkgs.openiscsi ];

However, NixOS doesn’t place the files in the same place as Longhorn expects (GitHub issue). Longhorn expects to find mount.nfs in the PATH, but in NixOS, it’s actually found in /nix/store/2l9hiinf01ikdjjxd1lafb9mqs5ssfp5-nfs-utils-2.7.1/bin/mount.nfs (the path may differ on your host). When it goes to run nsenter --mount=/host/proc/14084/ns/mnt --net=/host/proc/14084/ns/net mount.nfs in the container to break out of the container into the host namespaces and mount the NFS volume, it looks in the PATH of /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin and doesn’t find it so it fails.

We need to trick Longhorn into looking into the right place by overriding the PATH environment variable. When nsenter runs, it uses the PATH environment to invoke the command on the host OS. I like using Kyverno for this which is a Kubernetes program that can validate and mutate resources based on policies. I used it in the past and already had it installed on my cluster, so this next part worked well.

Inspired by this comment, I created a simple Kyverno policy that mutates every pod in the longhorn-system namespace to include our custom PATH.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: longhorn-add-nixos-path
 annotations:
 policies.kyverno.io/title: Add Environment Variables from ConfigMap
 policies.kyverno.io/subject: Pod
 policies.kyverno.io/category: Other
 policies.kyverno.io/description: >-
 Longhorn invokes executables on the host system, and needs
 to be aware of the host systems PATH. This modifies all
 deployments such that the PATH is explicitly set to support
 NixOS based systems.
spec:
 rules:
 - name: add-env-vars
 match:
 resources:
 kinds:
 - Pod
 namespaces:
 - longhorn-system
 mutate:
 patchStrategicMerge:
 spec:
 initContainers:
 - (name): "*"
 env:
 - name: PATH
 value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/run/wrappers/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
 containers:
 - (name): "*"
 env:
 - name: PATH
 value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/run/wrappers/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin

After that, I redeployed the Longhorn pods and the containers were able to start up without crashing.

iSCSI

However, trying to mount a volume on the host would fail:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


failed to execute: /usr/bin/nsenter [nsenter --mount=/host/proc/14084/ns/mnt --net=/host/proc/14084/ns/net iscsiadm -m discovery -t sendtargets -p 10.42.0.20], output , stderr
 iscsiadm: can't open iscsid.startup configuration file etc/iscsi/iscsid.conf\n
 iscsiadm: iscsid is not running. Could not start it up automatically using the startup command in the iscsid.conf iscsid.startup setting. Please check that the file exists or that your init scripts have started iscsid.
 iscsiadm: can not connect to iSCSI daemon (111)!
 iscsiadm: can't open iscsid.startup configuration file etc/iscsi/iscsid.conf
 iscsiadm: iscsid is not running. Could not start it up automatically using the startup command in the iscsid.conf iscsid.startup setting. Please check that the file exists or that your init scripts have started iscsid.
 iscsiadm: can not connect to iSCSI daemon (111)!
 iscsiadm: Cannot perform discovery. Initiatorname required.
 iscsiadm: Could not perform SendTargets discovery: could not connect to iscsid
 exit status 20

Nodes cleaned up for iqn.2019-10.io.longhorn:unifi-restore" func="iscsidev.(*Device).StartInitator" file="iscsi.go:168

msg="Failed to startup frontend" func="controller.(*Controller).startFrontend" file="control.go:489" error="failed to execute:
 /usr/bin/nsenter [nsenter --mount=/host/proc/14084/ns/mnt --net=/host/proc/14084/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:unifi-restore -o update -n node.session.err_timeo.abort_timeout -v 15], output , stderr
 iscsiadm: No records found
 exit status 21"
 ...

It can’t find the iscsid service provided by the iscsid.socket and iscsid.service systemd units. Just installing the package doesn’t enable the service. For that we need to add:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


 { config, lib, pkgs, ... }:
 {
 environment.systemPackages = with pkgs; [
 pkgs.nfs-utils
 pkgs.openiscsi
 ];

+ services.openiscsi = {
+ enable = true;
+ name = "iqn.2005-10.nixos:${config.networking.hostName}";
+ };
 }

Then redeploy the application pod.

I forgot the Firewall

At this point everything is deploying, but I can’t seem to access anything running on this host. Normally Docker and Kubernetes hijack the IPTables rules, but on NixOS they don’t. Easy fix:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


networking.firewall.allowedTCPPorts = [
 53
 # nginx
 80
 443
 # mail
 25
 110
 143
 465
 587
 993
 995

 7946 # flannel
 8285 # flannel
 8472 # flannel

 # Rancher UI
 8443 # I don't know if this is actually used
 32121

 # TODO: Lock these down to intra-node only
 2379 # etcd-client
 2380 # etcd-cluster
 6443 # kube-apiserver
 10250
 10254
 9100 # prom-node-export
];
networking.firewall.allowedUDPPorts = [
 53

 7946 # flannel
 8472 # flannel
];

Missing kubelet logs

Next, I had an issue where kubelet logs would not show any logs from any pods:

1
2
3


$ kubectl --context=local -n cattle-system logs helm-operation-hpq86
Defaulted container "helm" out of: helm, proxy, init-kubeconfig-volume (init)
failed to try resolving symlinks in path "/var/log/pods/cattle-system_helm-operation-hpq86_f63d99d0-ad39-478f-be8f-bdc1f4657d1d/helm/0.log": lstat /var/log/pods/cattle-system_helm-operation-hpq86_f63d99d0-ad39-478f-be8f-bdc1f4657d1d/helm/0.log: no such file or directory

Looking at the host, no files were present in /var/logs/pods and nothing was in /var/lib/docker/containers/{container}/*-json.log. The default logDriver in Nix for Docker is journald. I tried switching to local which didn’t work, but this StackOverflow post suggests it needs to be set to json-file.

Easy fix. I changed updated the logDriver and nixos-rebuild switch to restart the Docker daemon fixed my logs:

1
2
3
4


virtualisation.docker = {
 enable = true;
 logDriver = "json-file";
};

Conclusion

Thus far, I feel that Nix is powerful and I like having declarative config that can manage everything about the OS. I can define a series of configuration files and define per host flakes for my servers and computers and pick and choose which machines get what config.

But I don’t find the language itself very intuitive. I tried to create my own derivative, but ran into issues. Over time, I hope it gets better though and I’ll keep playing with it.

My financial data scraping system

Sun, 18 Aug 2024 00:00:00 +0000

In my Importing and cleaning my Mint transactions, I worked through loading, cleaning, and solving for transfers.

However, Mint and other financial scraping tools are not authoritative and don’t expose everything that the bank itself will provide. For example, Mint and Monarch don’t have detailed enough stock transaction and position data to identify cost basis, tax lots, and positions. Directly going to the bank can give me higher precision time stamps, scans of checks, merchant addresses, and other attributes.

Summary

I’ve been working on this project for the better part of 2024 at this point. Since none of my banks or brokerages provided any sort of obvious API to access my data, I created a few different syncing jobs that used Playwright, a web browser testing and automation framework, to login to my accounts and pull data. I then stored all that data in a custom PostgreSQL DB and load it into Firefly and Ghostfolio. I also wrote Apache Zeppelin notebooks to do ad-hoc analytics and testing of new algorithms.

With this new found data, I found it possible to do quick analytics. For example, I was able to easily write some Python to decide whether I was getting enough returns to justify whether or a credit card annual fee was worth it.

The Architecture

The architecture breaks down into a few different components. First, we have the banks and brokerages that I want to scrape. Then I use Monarch to scrape all banks, and I use my own Playwright based scrapers to pull data from the banks that I specifically wrote scrapers for. This all gets written into a few different PostgreSQL tables. The Postgres tables are there as a copy of the data because they contain more data that Firefly/Ghostfolio can store and provides a place for me to test new transfer solving algorithms and rebuild the Firefly/Ghostfolio databases as I make improvements.

Security

Security is paramount given that I have to store my own account credentials to log into my accounts. I didn’t want to store my passwords in a Kubernetes Secret which would have been too risky in case of a malicious actor. I opted to leverage Vault and OpenBao (an open-source Vault fork) which software-based secret storage system to store everything sensitive including the usernames, passwords, TOTP secrets (Time based one time passwords. These are codes that last for about 30 seconds), and cookie jars.

Why cookie jars? As a it turns out, banks get nervous when you keep logging in from fresh browsers each time. They’ll make you go through verification steps that may include an SMS, or a mobile app push notification. What I do is sign in once manually and approve the request, then every time my sync job runs, I restore the entire cookie jar into the browser, and save it afterwards. This makes it look like my scraping job is somewhat human.

All data in Vault is encrypted at rest and encrypted in transit, so no plain-text password is stored on disk. The decryption key is stored only in memory. If my server ever restarts, I have to re-enter the recovery key(s) to unlock the Vault database.

Vault is then configured to only trust certain Kubernetes namespaces to read/write secrets. If you want to see exactly how this is configured, read my Vault install post.

Why not scrape everything myself?

In a previous post, I did talk about some issues that I had with Monarch around their web application loading ad scripts here. This has not yet been fixed and I hope they do fix it, so it might seem unusual that I’m still using it. Unfortunately, I realized that I couldn’t develop scrapers for every single application. Some of my banks (e.g. Citi) required me to 2FA (two factor auth. A password is one factor, an extra code makes it 2FA) using SMS every single time I logged in which was impractical to automate. Monarch and their data providers (Plaid, MX, and Finicity) have the ability to do OAuth auth and API based communications with banks, but that’s not possible for me without compliance reviews.

Monarch handled the banks and credit cards, whereas I focused on the stock brokerages because Monarch did not going to scrape the data that I needed.

Institutions

Venmo

Venmo was comparatively easy vs the other sites. I did this one mostly as a prototype for a simple, non-stock scrape target. Their login page seems to use Recaptcha, but it never prompted me with a captcha. What’s nice is they have a hidden JSON API to transactions, but it’s wonky. My Playwright script will login to venmo.com, then navigate to Statements, find the Download CSV button, get the URL, modify to grab it in JSON format and the last 30 days, then download that data and update my SQL table.

Fidelity

This one consumed most of my time. During login, Fidelity would randomly require a 2FA verification if I ever changed from my server to my dev desktop even though I statically defined the User-Agent. My scraper jobs don’t handle 2FA very well yet. Scraping the data was painful because Fidelity’s website seems to be composed of several different web frameworks and engines.

Some data, like positions and activity, was available using CSV which I used when available, but positions data only contains stock level data, not at lot level.

Other data is returned using server-rendered HTML, some of it is returned using JSON APIs. Sometimes I couldn’t figure out how to replicate web requests and would have to click a button and sniff the requests using Playwrights expect_request mechanism. I had to use BeautifulSoup to parse HTML responses in other times.

The server-rendered HTML was problematic because it was designed for humans. For example, the tax lot table would contain icons for wash sales or include non-dates like the string Various and other random details that was hard to process programmatically.

Betterment

Betterment was another (not) fun scraping. The plus side is while they did require 2FA auth, they supported TOTP based auth. I stored the TOTP seed in my Vault DB and used pytotp to generate the token. I’ve since learned that Vault supports a TOTP provider natively which would be more secure because it doesn’t vend out the TOTP seed, it just vends out the OTP codes. That’s better because it reduces the risk of full exposure.

Once logged in, it was difficult to get the data I needed. They had some basic CSV exports, but it was severely lacking in useful data. The transaction level data CSV is unsuitable because it doesn’t contain symbol information. Their reports don’t even provide quarterly gains data making estimated tax payments hard.

I even email them every few years and they never implemented any of my requests. They even told me We are not able to provide API access to an individual, though as noted you can access information through programs such as TurboTax. Clearly they have APIs, they just don’t make them available.

The only way to get access to activity data was to scrape PDF files. Which, I being masochist, did just that. Did you know that PDFs don’t have a DOM like HTML? It’s just absolute positioning of content. How do you parse a table of content? Well, have to grab the (x, y) coordinates of text that I expect to appear above, then the disclaimers that appear below, grab all the text in between that and somehow make a table out of that.

Citibank

Citi was unworkable. They required an SMS OTP every single time I signed in. No option to remember it. While I could try to extract the OTP from the SMS’s on my phone using Home Assistant’s Android App sensor, I have not gone down that path given the fragility of that setup.

What did I learn?

Banks don’t make it easy to scrape your own data. They make it hard with 2FA authentication using push notifications or SMS (which is no-longer recommended by NIST). I also have to take care not to authenticate too often, which makes debugging difficult because I sometimes debug the script by running the sync job entirely which logs out and logs back in. My session saving only stores cookies and localStorage, but does not persist any session cookies or sessionStorage.

Synchronizing data without strong identifiers is tricky. When scraping data, I basically get a tuple of (date, amount, description). Without having some kind of identifier to uniquely and stably identify each transaction makes it challenging to figure out when a transaction should be created, updated, or deleted. This is similar to the problem I had when identifying transfer pairs. Instead I haveatch based on dates, amounts, and sometimes descriptions, but that often causes me to delete and recreate instead of updating. This isn’t a major problem because it self corrects, but annoying.

Where’s the code?

Right now, my code is pretty coupled to my system, however I’m slowly moving to my GitHub repo here: ajacques/own-your-finance.

What’s next?

Improve reliability - right now the scrapers don’t handle unexpected things well like 2FA, or slow or failed navigations. Next would be to implement a 2FA handler to get OTP codes
Handle more institutions - my goal is to support as many of the banks I or my friends use to reduce dependency on aggregators
Better analytics - right now Firefly and Ghostfolio are fine, but they lack in some of the features I’m looking for like good charts and tables

Securing data using Vault in a Home Lab

Mon, 12 Aug 2024 00:00:00 +0000

I have several projects running in my Home Lab that now have to store and use sensitive secrets. In my Self-hosted finances series, I developed software to scrape my own bank statements (more on that coming soon.) In other projects, I store API keys to manage DNS or even my dedicated servers.

These applications all run in Kubernetes, which does support Secrets, however, by default, they are not encrypted and are easily accessible to actors that have access to the K8s API.

It supports encryption at rest using a static encryption key or KMS plugin, which is fine, but I wanted to try out Hashicorp Vault which stored the decryption keys in memory only, so if somebody were to get access to my server (say by breaking into my house and stealing my server), they wouldn’t get access to all my secrets.

Pre-requisites

This post assumes that you have a working Kubernetes cluster.

Installation into Kubernetes

Instead of duplicating the guides that already exist, I recommend following this guide: https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-raft-deployment-guide

The plan

In my cluster, I have two actors:

Various Kubernetes pods - limited read/write to different buckets
Me - full admin access

Configure an Auth Method

Authentication Methods are the way actors like you or your services authenticate with Vault. Combined with policies, it defines what actions are allowed to be performed.

We’ll have two auth methods (mapping to the above actors):

Kubernetes
userpass

Create an admin policy

Let’s create a basic admin policy that will be assigned to yourself with full permissions. This policy has full admin access so it should only be used by trusted actors.

In the Vault UI, go to Policies > Create ACL Policy. Name it admin or similar and click create.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


path "sys/health"
{
 capabilities = ["read", "sudo"]
}

path "*"
{
 capabilities = ["create", "read", "update", "list", "sudo", "delete"]
}

path "sys/policies/acl"
{
 capabilities = ["list"]
}

path "data/*" {
 capabilities = ["read", "list", "update"]
}

path "sys/policies/acl/*"
{
 capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}


path "auth/*"
{
 capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

path "sys/auth/*"
{
 capabilities = ["create", "update", "delete", "sudo"]
}

path "sys/auth"
{
 capabilities = ["read"]
}


path "secret/*"
{
 capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

path "sys/mounts/*"
{
 capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

path "sys/mounts"
{
 capabilities = ["read"]
}

User/Password

It’s not a good practice to log in using the root key every time, so we’re going to create a username/password to use instead. To be pragmatic, I still have the root key because I know I’m going to need to make a change eventually. If this were a business, I’d lock it in a safe or destroy it.

I recommend enabling a Lease TTL which will cause the session to automatically expire.

Then create a user and attach the admin policy to the user.

Kubernetes Auth Method

The Kubernetes Auth method defines how Kubernetes pods will authenticate to the Vault server and be able to receive and update credentials. Create the Kubernetes auth method.

Then configure it to point to your apiserver. For example, the Kubernetes host might be https://10.43.0.1:443/ and the Kubernetes CA Certificate can be retrieved using:

1

kubectl get configmap kube-root-ca.crt -o jsonpath="{['data']['ca\.crt']}"

Getting application specific

Now that Vault is setup, we need to create the resources that pods will use. This will get into application specific configuration so you might do this one or more times. Vault uses the term engine to describe something where secrets are stored or vended. There’s a few different types, but for storing API keys, username/passwords to my accounts, a KV or (key-value) storage should be used. Create a KV engine and name it something meaningful for your application. For example, financialcreds.

Create the Vault Policies

Next, we need a policy that allows services to access it (we already have access through the admin policy).

Go back to Policies > Create ACL Policy and create a policy like below. Note how the path only gives access to the engine. You can also do something like foobar/baz/* to give access to a folder

1
2
3


path "financialcreds/*" {
 capabilities = ["read", "update", "list"]
}

Create Vault Roles

In Vault, a role maps the Kubernetes System Account to the list of Vault policies and permissions, and ultimately the secrets that the pod can access. Some useful configuration to specify:

Name: Role name. This will be used in your code later
Bound service account names: The name of the Kubernetes SystemAccount resource
Bound service account namespaces: The namespace(s) where the SystemAccount exists
Generated Token’s Maximum TTL: (optional) Sets the maximum lifetime the token lives for. This ensures that if a token leaks from a short-term job, it can’t be used for long. For my periodic scheduled sync jobs, I use 30m so credentials expire after 30 minutes.
Generated Token’s Policies: Specify the policies that this role has access to. Add the policy you just created in the previous section (not the admin one).
Generated Token’s Type: (optional) Vault docs

Using it in practice

If you did everything right, you should be able to run a job that authenticates to Vault and grabs a secret.

First we need a ServiceAccount with a matching name in the Vault role. Make sure to set automountServiceAccountToken: true.

1
2
3
4
5
6


apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
 name: my-account-specified-in-vault-role
 namespace: my-namespace-specified-in-vault-role

Then update the CronJob, Deployment, etc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: batch/v1
kind: CronJob
metadata:
 name: sync-job
 namespace: my-namespace-specified-in-vault-role
spec:
 jobTemplate:
 spec:
 template:
 spec:
 serviceAccountName: my-account-specified-in-vault-role

And some Python sample code to login and read:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


import hvac
import os
from hvac.api.auth_methods import Kubernetes

vaultUrl = 'http://vault.vault.svc.cluster.local.'
if 'KUBERNETES_SERVICE_HOST' in os.environ:
 client = hvac.Client(url=vaultUrl)
 jwt = open('/var/run/secrets/kubernetes.io/serviceaccount/token').read()

 Kubernetes(client.adapter).login(
 role="rolename", # Role name specified above
 jwt=jwt
 )
else:
 # 
 key = os.environ['VAULT_TOKEN']
 client = hvac.Client(url=vaultUrl, token=key)

Conclusion

We didn’t do anything store anything in Vault yet, but just setup the basics. In my future posts, I’ll be showing how I leverage Vault to manage my financial scraping.

Securing MQTT Traffic using cert-manager

Sat, 06 Jul 2024 00:00:00 +0000

I use MQTT in my home lab to connect different Home Lab services like ESPHome, Home Assistant, Node Red, etc. It’s great because it’s a light-weight way to decouple these services, but by default there’s no security. I can’t prevent a sensor from manipulating another sensor’s data, I can’t prevent somebody who has network access from monitoring messages.

In this post, I’m going to walk through enabling TLS with usernames and passwords or mTLS (Mutual TLS) using cert-manager. Cert-manager supports a mechanism to generate self-signed CA certs that I will use.

(Pre-req) MQTT Broker

I’m using Mosquitto as my MQTT broker and will assume you already have it setup. Additionally, you’ll need to have the ability to edit the configuration files. I created a Kubernetes PVC and mounted it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70


apiVersion: apps/v1
kind: Deployment
metadata:
 name: mqtt-broker
 namespace: smarthome
spec:
 replicas: 1
 strategy:
 rollingUpdate:
 maxSurge: 0%
 maxUnavailable: 100%
 type: RollingUpdate
 selector:
 matchLabels:
 workload.user.cattle.io/workloadselector: deployment-smarthome-mqtt-broker
 template:
 metadata:
 labels:
 workload.user.cattle.io/workloadselector: deployment-smarthome-mqtt-broker
 spec:
 containers:
 - image: eclipse-mosquitto:2.0.18-openssl
 imagePullPolicy: IfNotPresent
 name: mqtt-broker
 resources:
 limits:
 memory: 16Mi
 requests:
 cpu: 5m
 memory: 16Mi
 terminationMessagePath: /dev/termination-log
 terminationMessagePolicy: File
 volumeMounts:
 - mountPath: /mosquitto/config/
 name: config
 readOnly: true
 # Side-car will trigger Mosquitto to reload when the config changes
 # No need to restart the entire pod.
 - env:
 - name: CONFIG_DIR
 value: >-
 /config/mosquitto.conf,/config/acl.conf,/config/passwd
 - name: PROCESS_NAME
 value: mosquitto
 image: ajacques/config-reloader-sidecar:latest
 imagePullPolicy: IfNotPresent
 name: config-reloader
 resources:
 limits:
 memory: 16Mi
 requests:
 memory: 16Mi
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 add:
 - KILL
 drop:
 - ALL
 privileged: false
 readOnlyRootFilesystem: true
 runAsNonRoot: false
 volumeMounts:
 - mountPath: /config/
 name: config
 readOnly: true
 volumes:
 - name: config
 persistentVolumeClaim:
 claimName: mqtt-broker

Setting up the private CA

First step is to create a root certificate that will serve as the trusted root store

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: smarthome-ca
spec:
 selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: smarthome-ca-cert
 namespace: smarthome
spec:
 commonName: homelab-ca
 duration: 87600h0m0s
 isCA: true
 issuerRef:
 group: cert-manager.io
 kind: Issuer
 name: smarthome-ca
 privateKey:
 algorithm: ECDSA
 size: 256
 secretName: root-secret

CA Signer

Next step is to create a cert-manager Issuer that will sign certificates using the root CA created above:

1
2
3
4
5
6
7
8


apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: smarthome-issuer
 namespace: smarthome
spec:
 ca:
 secretName: root-secret

MQTT Server Certificate

Next step is to create a certificate for the MQTT server. Note that I include both the external and internal Kubernetes DNS names as a Subject Alternate Name. This ensures that the certificate will validate both inside the cluster and outside.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: mqtt-server
 namespace: smarthome
spec:
 commonName: mqtt.example.com
 dnsNames:
 # Include the internal domain name too if
 # services are directly connecting to it
 - mqtt.example.com
 - mqtt-headless.smarthome.svc.cluster.local
 issuerRef:
 group: cert-manager.io
 kind: Issuer
 name: smarthome-issuer
 privateKey:
 algorithm: ECDSA
 size: 256
 secretName: mqtt-server-cert

Configuring the server

Right now, the MQTT broker is going to have all these identities, but is going to do nothing with them. Combining this with some authorization rules using ACLs will enable us to control what topics each device can read from and write to.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76


 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: mqtt-broker
 namespace: smarthome
 spec:
 replicas: 1
 selector:
 matchLabels:
 workload.user.cattle.io/workloadselector: deployment-smarthome-mqtt-broker
 template:
 metadata:
 labels:
 workload.user.cattle.io/workloadselector: deployment-smarthome-mqtt-broker
 spec:
 containers:
 - image: eclipse-mosquitto:2.0.18-openssl
 imagePullPolicy: IfNotPresent
 name: mqtt-broker
 resources:
 limits:
 memory: 16Mi
 requests:
 cpu: 5m
 memory: 16Mi
 terminationMessagePath: /dev/termination-log
 terminationMessagePolicy: File
 volumeMounts:
 - mountPath: /mosquitto/config/
 name: config
 readOnly: true
+ - mountPath: /ssl/cert
+ name: ssl
+ readOnly: true
 - env:
 - name: CONFIG_DIR
- value: /config/mosquitto.conf,/config/acl.conf,/config/passwd
+ value: /config/mosquitto.conf,/config/acl.conf,/config/passwd,/ssl/tls.key,/ssl/tls.crt
 - name: PROCESS_NAME
 value: mosquitto
 image: ajacques/config-reloader-sidecar:latest
 imagePullPolicy: IfNotPresent
 name: config-reloader
 resources:
 limits:
 memory: 16Mi
 requests:
 memory: 16Mi
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 add:
 - KILL
 drop:
 - ALL
 privileged: false
 readOnlyRootFilesystem: true
 runAsNonRoot: false
 terminationMessagePath: /dev/termination-log
 terminationMessagePolicy: File
 volumeMounts:
 - mountPath: /config/
 name: config
 readOnly: true
+ - mountPath: /ssl
+ name: ssl
+ readOnly: true
 volumes:
 - name: config
 persistentVolumeClaim:
 claimName: mqtt-broker
+ - name: ssl
+ secret:
+ defaultMode: 420
+ optional: false
+ secretName: mqtt-server-cert

To start, we have a mosquitto.conf that looks like the below. It means that anybody can connect on port 1883 with no auth.

1
2
3
4
5
6


log_dest stdout
log_type notice
per_listener_settings true

listener 1883
allow_anonymous true

I’ll add a new port that requires authentication. The first port requires username and passwords (Make sure to create a passwd file if you want this.) Some devices, like esp devices running esphome don’t have great support for mTLS, so I have to use usernames.

1
2
3
4


# TLS but with usernames and passwords
listener 8882
tls_version tlsv1.2
password_file /mosquitto/config/passwd

The next port requires mTLS for all connections. This will be added in parallel so I can slowly move devices over:

1
2
3
4
5
6
7
8


# Mutual TLS - Encryption
listener 8883
tls_version tlsv1.2
require_certificate true
use_identity_as_username true
cafile /ssl/cert/ca.crt
certfile /ssl/cert/tls.crt
keyfile /ssl/cert/tls.key

Configuring ACLs

In the server configuration, we need to add:

1
2
3
4
5


 listener 8882
+acl_file /mosquitto/config/acl.conf

 listener 8883
+acl_file /mosquitto/config/acl.conf

You can read more about the ACL file format here. My file looks like this. By default authenticated devices are able to read and write to their own device topic. Each authenticated device has a username. For username/password, it’s obviously the username and for mTLS it comes from the CN in the subject (You can see an example in the Usage section below.)

This first section gives a default. All devices have access to their own topics ({device}/{sensor_name}) and their equivalent Home Assistant discovery topics (homeassistant/{domain}/{device}/{sensor_name}). This is implemented with the %u username placeholder. Most of my devices are running esphome so this works well.

1
2
3


pattern readwrite esphome/discover/%u
pattern readwrite %u/#
pattern readwrite homeassistant/+/%u/#

Some clients need different privileges because they aren’t esp devices. For example, my control software needs higher privileges. My Node-Red service can consume anything and write to anything (I should restrict this more) and my AppDaemon has some access.

1
2
3
4
5
6
7


user node-red
topic read #
topic write #

user appdaemon
topic read #
topic write homeassistant/#

Place this acl.conf in the same folder as the configuration file.

Usage

MQTT Client Certificates

Repeat this step for as many clients that you have.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: mqtt-{clientname}
 namespace: smarthome
spec:
 commonName: {clientname}
 issuerRef:
 group: cert-manager.io
 kind: Issuer
 name: smarthome-issuer
 privateKey:
 algorithm: ECDSA
 size: 256
 secretName: mqtt-{clientname}-cert

This certificate can then be mounted into any Kubernetes pod and used.

Node-Red example

For example, in Node-Red I’ll do:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: hass-node-red
   namespace: smarthome
 spec:
     spec:
       containers:
           name: node-red
           volumeMounts:
+            - mountPath: /ssl/mqtt
+              name: mqtt-ssl
+              readOnly: true
       volumes:
+        - name: mqtt-ssl
+          secret:
+            defaultMode: 420
+            optional: false
+            secretName: mqtt-node-red-cert

And configured in the UI to use this here:

ESPHome

As of the time of writing this, I didn’t have great luck with mTLS on my esp32 devices. First, I had to extend the lifetime of the certificates so I didn’t have to reflash devices every time they expired. It was also hard to get it to verify the Host identity too because the mqtt.certificate_authority field is only available on platform: esp-idf, and not all my device configurations worked with this (e.g. NeoPixels didn’t work with platform: esp-idf).

Usernames

However, it does support have support for username/passwords. In the above configuration, I add a password listener on port 8882, so this can be done using:

1
2
3
4
5


mqtt:
 broker: mqtt.home.ajacqu.es
 port: 8882
 username: {myusername}
 password: {mypassword}

To enable CA verification on esp32 and esp-idf devices, we need to import the certificate authority which is available in the ca.crt key of the Secret generated by cert-manager.

Either copy the ca.crt value out of the secret that cert-manager generates, or inject it into ESPHome:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: esphome
 namespace: smarthome
 spec:
 spec:
 containers:
 - name: main
 volumeMounts:
 - mountPath: /config
 name: data
 - mountPath: /config/.esphome/build
 name: tempfolder
 subPath: build
 - mountPath: /config/.esphome/platformio
 name: tempfolder
 subPath: platformio
 - mountPath: /config/.esphome/external_components
 name: tempfolder
 subPath: external_components
+ - mountPath: /config/ssl-ca.crt
+ name: tls-cert
+ readOnly: true
+ subPath: ca.crt
 volumes:
 - hostPath:
 path: /tmp/k8s-esphome/
 type: DirectoryOrCreate
 name: tempfolder
 - name: data
 persistentVolumeClaim:
 claimName: esphome2
+ - name: tls-cert
+ secret:
+ defaultMode: 420
+ optional: false
+ secretName: mqtt-server-cert

Then use it in any device:

1
2
3
4
5
6


mqtt:
 broker: mqtt.example.com
 port: 8882
 username: xyz
 password: abc
 certificate_authority: !include ssl-ca.crt

Conclusion

Unfortunately, I went through this exercise and don’t have a great story for mTLS on the ESPHome devices which is an important use case, but I did get it working on my internal services, the ones that have higher privileges anyway.

Auto switch between light and dark mode on GNOME

Tue, 18 Jun 2024 00:00:00 +0000

I recently got a Framework laptop and installed Ubuntu on it to give Linux for laptops a chance after using Windows and Mac for work for years. One thing I wanted was to be able to switch between light mode and dark mode automatically depending on the time of day. GNOME had a blue-light filter mode that could automatically turn on, but it didn’t appear to have a way to switch between light mode and dark mode at the same time.

The GNOME settings window showing the ability to automatically enable/disable a blue light filter at specific times of the day.

Research

Sort of Working

The following command seemed to switch the theme, but not all apps would follow. The terminal and GNOME would follow, but Firefox wouldn’t.

1
2


gsettings set org.gnome.desktop.interface gtk-theme Yaru-dark
gsettings set org.gnome.desktop.interface gtk-theme Yaru

Digging through Code

The GNOME settings window showing light and dark mode, but no option to schedule it

To figure this out, I dove into the source code for the GNOME settings tool (GitHub GNOME/gnome-control-center). Searching for strings like “light” and “dark” led me to the file panels/background/cc-background-panel.c which implemented this settings page.

These methods suggested there was a different setting that controlled the light/dark mode:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


static void
on_color_scheme_toggle_active_cb (CcBackgroundPanel *self)
{
 if (gtk_toggle_button_get_active (self->default_toggle))
 set_color_scheme (self, G_DESKTOP_COLOR_SCHEME_DEFAULT);
 else if (gtk_toggle_button_get_active (self->dark_toggle))
 set_color_scheme (self, G_DESKTOP_COLOR_SCHEME_PREFER_DARK);
}

static void
set_color_scheme (CcBackgroundPanel *self,
 GDesktopColorScheme color_scheme)
{
 // Removed

 g_settings_set_enum (self->interface_settings,
 INTERFACE_COLOR_SCHEME_KEY,
 color_scheme);
}

Then using the following commands it actually worked and fully changed the color in Firefox and across my desktop environment.

1
2


gsettings set org.gnome.desktop.interface color-scheme prefer-dark
gsettings set org.gnome.desktop.interface color-scheme prefer-light

Scripts

I started with the script from here and changed the gsettings key

/usr/bin/gnome-theme-switcher.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


#!/bin/bash
set_theme() {
 if [[ "$1" == "dark" ]]; then
 new_gtk_theme="prefer-dark"
 elif [[ "$1" == "light" ]]; then
 new_gtk_theme="prefer-light"
 else
 echo "[!] Unsupported theme: $1"
 return
 fi

 export DBUS_SESSION_BUS_ADDRESS=$DBUS_SESSION_BUS_ADDRESS
 echo "$DBUS_SESSION_BUS_ADDRESS"
 current_gtk_theme=$(gsettings get org.gnome.desktop.interface color-scheme)
 # echo "[.] Currently using ${current_gtk_theme}"
 if [[ "${current_gtk_theme}" == "'${new_gtk_theme}'" ]]; then
 echo "[i] Already using gtk '${new_gtk_theme}' theme"
 else
 echo "[-] Setting gtk theme to ${new_gtk_theme}"
 gsettings set org.gnome.desktop.interface color-scheme ${new_gtk_theme}
 echo "[✓] gtk theme changed to ${new_gtk_theme}"
 fi
}

# If script run without argument
if [[ -z "$1" ]]; then
 currenttime=$(date +%H:%M)
 if [[ "$currenttime" > "17:00" || "$currenttime" < "09:00" ]]; then
 set_theme dark
 else
 set_theme light
 fi
else
 set_theme $1
fi

Move the file into a safe location. I use /usr/bin/ so it’s not user-writable by default:

1
2


sudo mv ~/gnome-theme-switcher.sh /usr/bin/
sudo chmod +x /usr/bin/gnome-theme-switcher.sh

Scheduling using systemd

Next up is getting systemd to run the command. The service unit defines what command to run.

~/.config/systemd/user/auto-theme-switcher.service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[Unit]
Description=Auto adjusts GNOME theme between dark and light to match the time
After=suspend.target

[Service]
Type=oneshot
ExecStart=/usr/bin/gnome-theme-switcher.sh
TimeoutSec=30
StandardOutput=journal
PrivateTmp=true
ProtectSystem=full
ProtectHome=read-only

[Install]
WantedBy=default.target

The timer defines when the command should run. I choose to run mine at 8pm and 9am. If you change it, make sure to update the script above.

~/.config/systemd/user/auto-theme-switcher.timer:

1
2
3
4
5
6
7
8
9


[Unit]
Description=Auto adjusts GNOME theme between dark and light to match the time

[Timer]
OnCalendar=*-*-* 20:00:00 # 8pm
OnCalendar=*-*-* 09:00:00 # 9am

[Install]
WantedBy=timers.target

Then run the following to enable it

1
2


systemctl --user enable auto-theme.service
systemctl --user enable auto-theme.timer

You can test the command by running:

1

systemctl --user start auto-theme.service

Update: Nov 11, 2025

I’ve since switched to KDE which has a built in night mode, so I don’t have to use this. However, a reader helpfully shared their update to the script here which incorporates automatic sunset/sunrise schedule adjusting

References

Publishing to Firefly-iii using Pandas

Sat, 08 Jun 2024 00:00:00 +0000

Previously, in my Self-hosted finances series, I cleaned and identified transfers in my Mint transactions for the purposes of of importing into Firefly-iii. In this post, I’m going to import the transactions into Firefly-iii.

This part is comparatively easy vs the previous steps, however it’s only a one time import. A continue updating workflow is tricky and I’m working on some logic to do that.

Authenticating

First, we need a token that allows us to call the API. To get this, login to Firefly, and open Options, then Profile.

Then go to OAuth and click Create a new token

Firefly will then show you a token. Copy and store this in the script as a variable and also specify the endpoint where Firefly is being served:

1
2


fireflyToken = 'eYJ...'
host = 'https://firefly.example.com'

Constructing the request

This next method will translate the DataFrame created in the previous post request into the format that the Firefly API expects and creates the transaction.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


import json
import requests

session = requests.Session()

def send_to_firefly(record):
 item = {
 'type': record['type'],
 'amount': record['amount']
 }
 for key in ['description', 'destination_name', 'source_name', 'category_name', 'notes']:
 if key in record and not pd.isnull(record[key]):
 item[key] = record[key]
 for key in ['source_id', 'destination_id']:
 if key in record and not pd.isnull(record[key]) and record[key] != '':
 item[key] = str(int(record[key]))
 item['foreign_amount'] = '0'
 item['reconciled'] = False
 item['order'] = '0'
 # If you use something other than USD, change this value
 item['currency_id'] = '8'
 item['currency_code'] = 'USD'
 for key in ['date', 'process_date']:
 if key in record and not pd.isnull(record[key]) and record[key] != '':
 item[key] = record[key].isoformat()
 if isinstance(record['tags'], str):
 item['tags'] = [record['tags']]
 else:
 item['tags'] = []
 payload = {
 "transactions": [
 item
 ],
 "apply_rules": True,
 "fire_webhooks": False,
 "error_if_duplicate_hash": False
 }
 try:
 json.dumps(payload)
 except Exception as e:
 raise Exception(f"Can't serialize '{item}: {e}")
 resp = session.post(f"{host}/api/v1/transactions", headers={"Authorization": f"Bearer {fireflyToken}", "Content-Type": "application/json", 'Accept': 'application/json'}, json=payload, allow_redirects=False)
 if resp.status_code == 200:
 return pd.Series([resp.status_code, resp.json(), resp.json()['data']['id']], index=['status', 'message', 'firefly_id'])
 else:
 return pd.Series([resp.status_code, resp.json()], index=['status', 'message'])

Sending the request

Next, it’s easy to send the requests to Firefly:

1
2
3
4
5


# First find the transfers and translate into common format
output = transactions.apply(func=process_record, axis=1, result_type='expand')

# Then send it to Firefly
fireflyResult = output[output['amount'] != 0].apply(func=send_to_firefly, axis=1, result_type='expand')

Reviewing Results

Firefly can return non-200 status code responses. Running this will identify all the non-successful writes. In theory, this should empty. It’s a good idea to check the error messages and see what’s wrong.

1

print(fireflyResult[(fireflyResult['status'] != 200)])

Conclusion

Once we’ve identified the transfers, it’s comparatively easy to write the one-time extract to Firefly. The code can be found in my GitHub repository. I’ve got a few projects in progress right now, including tools to automatically scrape data from accounts and how to merge new transactions into Firefly.

GoDaddy is now blocking API access

Tue, 04 Jun 2024 00:00:00 +0000

I own few domains and one of those domains is registered at GoDaddy. This is for historical reasons because this domain is on the .es TLD but my preferred registrar, PorkBun or CloudFlare, do not support this TLD. I kept it there mainly because I’ve had it for 10+ years and there were some new identify requirements that I didn’t want to deal with yet.

I use external-dns as a tool to automatically to take my Kubernetes Ingress resources and register them in my DNS zone. This tool would use the GoDaddy API to create, edit, and delete the DNS records automatically for me and point them to my own servers. Sure, I could do this myself because I don’t have that many records, but automation is fun.

Then today, I’m greeted with this error from external-dns and it just crashing and restarting:

1

level=fatal msg="Error ACCESS_DENIED: \"Authenticated user is not allowed access\""

Some web searching, led me to a few different posts by others complaining and this response from GoDaddy Support:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


Hi,

Thank you for reaching out to us regarding the recent changes to our Domain API.

We wanted to inform you that we have recently updated our Domain API requirements. As part of this update, customers are now required to have 50 or more domains in their account to utilize the API. Unfortunately, as you currently only have 1 domain in your account, access to the API is blocked for you.

However, we want to assure you that you still have access to the OTE API without any blocks.

We apologize for any confusion or inconvenience this may have caused. If you have any further questions or need assistance with any other aspect of our services, please don't hesitate to reach out.

Thank you for your understanding.

Regards,

API Support Team

I have less than 10 domains in my account, so they decided that they would block my API access like others. No warning at all was given to me over email, this wasn’t clearly indicated in the developer console either. Corporate greed. Now I have to figure out if I want to transfer it or just manage the entire DNS zone myself, which I already do for technowizardry.net.

The confusing world of scraping my own stock portfolio

Wed, 22 May 2024 00:00:00 +0000

Over the past few months, as part of my self hosted finances series I’ve been working to extract all of my stock portfolio into some kind of self hosted database. I came across Ghostfolio, which is an open-source (with a paid hosted edition) tool for tracking stock portfolios. It was able to give me a portfolio view across multiple brokerages, automatically fetched stock prices, and gave some basic allocation reporting.

The problem

Ghostfolio doesn’t have any mechanism of actually knowing about your transactions and I don’t want to manually do it because that takes time and can have mistakes. Brokerages don’t expose this. Options like Plaid cost money and require extensive compliance requirements and don’t even provide all data that I need.

While some of banks provided CSV exports, that was rarely sufficient. That spawned a several month long project to see if I could build my own scraping to import it into Ghostfolio and centralize all my financial data into one place so I could perform analytics on it.

This led me on a multi month long project ot build tooling to synchronize my stock positions into Ghostfolio. I ended up building several web browser automation tools to just scrape what I need, however this quickly became a giant mess requiring a lot of work to figure out. I thought that working with bank and card transactions was hard, but stocks introduce an entirely new set of challenges.

In this post, I’m not even going to talk about how I implemented my scraping system, just the horrors I faced.

Terminology and Concepts

Before we can talk about complexity, let’s break down the concepts.

Portfolio - The full set of current positions
Current Position - The current # of shares of a given stock. This is the minimum information we need
Cost Basis - The purchase price for all shares (now we have the profit)
Stock Lot - A lot is a block of shares purchased on a given date and time with a given purchase price. Combined with cost basis, this gives us some useful insights for tax planning
Realized gains/losses - A purchase and sale of a given stock lot
Historical Activity - The history of buy, sell, and dividends. With this we now know how our returns over a given period of time

With just the positions, you can only know your account value. Adding cost basis means we now know the profit. Knowing the lots means you can now make tax planning decisions (for example is it better sell FIFO or the lot with the highest basis). Once you know the historical activity, you can now know the value of your account over time. Realized gains/losses gives you the historical profit.

What does Ghostfolio need?

Ghostfolio processes in terms of the buy, sell, dividend activity. i.e. I bought x shares on y date for z monies. Thus, the historical activity should be the best way to import data.

Limited activity data

Sounds easy, right? Whelp, unfortunately not every brokerage gives this information in this manner. Fidelity provides a CSV export of the activity, but only goes back 5 years. What happens if you have shares that are older than that? Using just the activity log, you’ll end up short in your portfolio.

An incorrect short-position in Ghostfolio caused by the missing BUY side of the transaction because it’s >5 years old:

Betterment provides a dividends in CSV, lot level cost basis in CSV, and activity as a PDF. This is even more difficult because now you can’t identify any stock sales meaning that your portfolio incorrectly inflates upwards. In addition, it’ll incorrectly calculate historical account balances and assume you invested more money in later than you actually did. I request them to provide some useful every few years and they regularly say they don’t support it. Even though they clearly offer it through third-party aggregators like Plaid.

So here we have two different brokerages that provide completely different data offerings and none of them are actually usable.

Inferring it from other sources

Instead of looking just at one thing, the activity log, can we try to combine all the information the broker does provide us to infer the activity going back as far as possible?

For example, Fidelity has lot level realized gains going back 9 years, the current positions, open stock lots, in addition to the activity log. The current positions has a CSV download, but it only gives me a cost basis and current value, but I can’t figure out a BUY because I don’t know when that was purchased.

By open stock lots

The unrealized stock lots are available in the UI, but are server-side rendered as HTML (no friendly JSON or CSV) so I have to write a custom parser. Here’s the data it provides:

Acquired	Term	$ Total Gain/Loss	% Total Gain/Loss	Current Value	Quantity	Average Cost Basis	Cost Basis Total
May-22-2015	Long	$100.00	+100%	$150	5	$10	$50
Jan-1-2024	Short	$5	+5%	$10	1	$5	$5

Is that enough to tell Ghostfolio there was a BUY on May 22, 2015? Sort of, but there’s a foot-gun. It’s entirely possible to buy 10 shares of a stock and then sell 5 from that lot, so you’ve split the original purchase. If I have activity going back that includes the 2024 share purchase above, I have to recognize this and not create duplicate purchase records in Ghostfolio. I accidentally made this mistake before and then inflated my Ghostfolio.

By PDF statement

Fidelity offers PDF statements going back 10 years and Betterment provides activity events as PDF too. However, PDF is a terrible format to parse. There’s no DOM structuring like HTML. I adopted to use a Python package called PDFQuery to parse the PDFs and found out that you basically have to use bounding boxes to select text based on where it appears on the page. For example, in one brokerage, the doc looks like:

Investment Details
A table detailing the trades
A bunch of legal disclaimers and notices

And to extract the table, I have to first find the text immediately before and after, then find those (x, y) coordinates, then find all the text in between those two lines and extract the table. Let’s hope they never change their disclaimer.

Even worse, institutions can and have regularly changed their page layouts over the years, so now you have to implement parsers depending on the years.

By realized gains

In Fidelity, I tried current positions and I tried the activity log and still got only partial data. What about the realized gains (i.e. closed positions on the UI)?

With this, we can see buy and sell transactions between the 5 and 10 year time range, but of course it has the same problem as the open tax lots, in that you have to ignore any transaction that you do have activity for to avoid a duplicate buy/sell. But of course this is only available as an entirely custom HTML table which requires browser automation to open the tabs and grab the table contents.

Rounding Bugs

Brokerage exports can give you information per share or total for the entire transaction. Ghostfolio takes the # of shares and the per share price, then multiplies number x share price = total price. If a data source gives me the transaction cost basis or value, then I have to do value / # of shares to get the per share price. That introduces error, especially if I try to match transactions.

For example, Betterment’s cost basis CSV export has:

Shares	Cost Basis
0.130046	13.81
0.131024	13.91

And the equivalent statements show:

Statement	Price	Shares	Value
#1	$106.27	0.130	$13.82
#2	$106.24	0.131	$13.92

We can clearly see that $13.81 is not equal to $13.82 or $13.92, so there’s clearly a difference in rounding in the CSV and the PDF statement. In addition, $13.81 / 0.130046 = 106.19319… which is not equal to $106.27 or $106.24.

Conclusion

In theory, if I combine all these different signals, I should be able reverse engineer most of the activity log at least for Fidelity, but good luck doing that across every institution and these institutions aren’t very interested in making this any easier. I email Betterment about some improvements that would make tax filing and extracting easier and each time they say not supported. Maybe I’ll rethink paying their management fee.

Solving for bank transfers using Pandas

Wed, 01 May 2024 00:00:00 +0000

Previously in Part 1, I talked about how to clean-up the transaction data from Mint to remove duplicates and add any missing transactions.

Solving for transfers

The next phase is to solve for the transfer pairs. A transfer pair is defined with a matching credit and debit transaction on two different accounts. In Firefly, a transfer is treated separately than a credit/debit because it’s excluded from the expense and income reports.

This ended up being trickier than I expected and I went through multiple phases. First attempt, I used Pandas merge_asof, but this did not handle multiple transactions around the same time. Then I developed a new method that ignored previously used transactions. This worked better, but it still struggled with complex transaction chains. For example, if I transferred $1k from account A to B, then from C to D, it might accidentally book a transfer from A to D and C to B which is incorrect.

The basic algorithm

Here we have the first iteration. How it works is for every single transaction, it looks forward 5 days, finds all transactions that are the opposite type (credit -> debit or vice versa) and the same amount, then tags the first one as the opposite. For example, in the below table, it’d identify the two $300 transactions as a pair.

Date	Description	Amount	Type	Account
1/1/2024	Transfer to XYZ	$300.00	debit	ABC
1/1/2024	Bought Avocado Toast	$4.99	debit	Credit Card
1/2/2024	Transfer from ABC	$300.00	credit	XYZ

The code looked like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95


def find_transfer(row, df, time_window_days=5):
 start_date = row['Date']
 end_date = start_date + timedelta(days=time_window_days)

 opposing_filter = (
 (df['Account Name'] != row['Account Name']) &
 (df['Date'] >= start_date) &
 (df['Date'] <= end_date) &
 (df['Amount'] == row['Amount']) &
 (df['Transaction Type'] != row['Transaction Type']) &
 (~df['Considered'])
 )

 transfer_pair = df[opposing_filter]
 if not transfer_pair.empty:
 df.at[row.name, 'Considered'] = True
 opposite = transfer_pair.iloc[0]
 return opposite
 else:
 return None


def process_record(row, df):
 if df.at[row.name, 'Considered']:
 return None

 pair = find_transfer(row, df)
 if pair is not None:
 df.at[pair.name, 'Considered'] = True
 notes = [
 row['Notes'],
 pair['Notes'],
 ]
 labels = list(filter(lambda x: isinstance(x, str) and x != '', [
 row['Labels'],
 pair['Labels']
 ]))
 if row['Transaction Type'] == 'debit':
 # Bank is on the left
 date = row['Date']
 process_date = pair['Date']
 notes.append(pair['Original Description'])
 return {
 'type': 'transfer',
 'date': row['Date'],
 'process_date': pair['Date'],
 'amount': row['Amount'],
 'category': pair['Category'],
 'description': row['Original Description'],
 'source_id': row['AccountId'],
 'destination_id': pair['AccountId'],
 'tags': labels,
 'format': 'transfer_debit',
 'notes': '\n'.join(filter(lambda x: isinstance(x, str) and x != '', notes))
 }
 else:
 date = pair['Date']
 process_date = row['Date']
 notes.append(row['Original Description'])
 return {
 'type': 'transfer',
 'date': date,
 'process_date': process_date,
 'amount': row['Amount'],
 'category': pair['Category'],
 'description': pair['Original Description'],
 'source_id': pair['AccountId'],
 'destination_id': row['AccountId'],
 'tags': labels,
 'format': 'transfer_credit',
 'notes': '\n'.join(filter(lambda x: isinstance(x, str) and x != '', notes))
 }

 result = {
 'date': row['Date'],
 'description': row['Original Description'],
 'amount': row['Amount'],
 'category': row['Category'],
 'tags': row['Labels'],
 'notes': row['Notes']
 }

 if row['Transaction Type'] == 'credit':
 result['type'] = 'deposit'
 result['destination_id'] = row['AccountId']
 result['source_name'] = row['Description']
 else:
 result['source_id'] = row['AccountId']
 result['destination_name'] = row['Description']
 result['type'] = 'withdrawal'

 return result

transactions['Considered'] = False
transactions.apply(func=process_record, axis=1, result_type='expand')

However, it made a lot of mistakes. For example, it would classify the following as a transfer from Venmo to the Credit Card. Which maybe one could say it’s a “transfer”, but I didn’t want that because the credit card would get paid by my checking account.

Date	Description	Amount	Type	Account
1/1/2024	Bought Dinner for my friends	$50	debit	Credit Card
1/1/2024	Friend paid me back	$50	credit	Venmo

Eventually, I extended that algorithm with rules based on how my accounts behaved. Let’s look at some of those options.

Not all transactions are transfers

Not all transactions are transfers, but sometimes it lines up that it looks like it’s a transfer because there’s a matching side. Just like the previous example. We can define some rules that match transactions that we know are not transfers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


def account(num: Union[int, List[int]]):
 if type(num) == int:
 return transactions['AccountId'] == num
 else:
 return transactions['AccountId'].isin(num)

def orig_descr_contains(msg: str):
 return transactions['Original Description'].str.contains(msg, case=False)

# List of all Firefly account ids that are credit cards
credit_cards = [5, 10, 15, 20]

brokerages = []

account_id_401k = 25

not_a_transfer = [
 # Maybe exclude an entire account. For example, your 401k may never have
 # a transfer (unless you take a loan or do a Mega Backdoor Roth)
 account(account_id_401k),

 # Or you want to exclude reimbursements via Venmo
 account(23) & orig_descr_contains('John Doe Paid '), # Venmo
 account(23) & ~transactions['Category'].isna() & (transactions['Category'] != 'Transfer'), # Venmos frequently appear like transfers because a friend reimburses me

 # Credit Card Payments (to the card) can be transfers, but
 # maybe they're never transfers going out (unless you do a Balance Transfer)
 account(credit_cards) & (transactions['Transaction Type'] == 'debit'),

 # A paycheck
 # While it is a transfer from your employer to you, without the
 # debit side, the algorithm could incorrectly classify it as a transfer
 orig_descr_contains('PAYROLL') | (transactions['Category'] == 'Paycheck'),

 # Interest paid is not a transfer
 orig_descr_contains('INTEREST PAYMENT'),
 transactions['Category'] == 'Interest Income',

 # Dividend reinvestment are not transfers
 account(brokerages) & (orig_descr_contains('DIVDEND') | orig_descr_contains('REINVESTMENT')),

 # Some accounts have $0 transactions as part of bookkeeping
 # They are not transfers
 (transactions['Amount'] == 0)
]

Then we can integrate it into the algorithm. While searching for new transactions, it excludes anything that is not considered relevant.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


# Compute the exclusion set
df_filter_relevant = ~reduce(lambda x, y: x | y, not_a_transfer)

def find_transfer(row, df, time_window_days=5):
 # ...
 opposing_filter = (
 (df['Account Name'] != row['Account Name'])
 & (df['Date'] >= start_date)
 & (df['Date'] <= end_date)
 & (df['Amount'] == row['Amount'])
 & (df['Transaction Type'] != row['Transaction Type'])
 & (~df['Considered'])
 & df_filter_relevant
 )
 # ...

def process_record(row, df):
 if df.at[row.name, 'Considered']:
 return None

 if df_relevant.loc[row.name]:
 transfer_pair = find_transfer(row, df)
 # ...

# ...

output = transactions.apply(func=process_record, axis=1, result_type='expand')

Figuring out the rules is tricky. What I do is to actually look at both sides of the transfers in the notes:

1

output[output['type'] == 'transfer']

And see what makes sense and what doesn’t. If you can’t figure out how to define a rule that ignores that transaction, then go back to your CSV file, add a new column called IsTransfer and put true or false into rows that the algorithm struggles with (you don’t have to annotate every row.) Then reload the file and add a rule:

1
2
3
4


not_a_transfer = [
 # ...
 ~transactions['IsTransfer']
]

Heuristic: Account Id in Description

The next problem is that if I make transfers with the same amount during the same time period, it can get confused and depending on the ordering in the transactions CSV, mis-attribute the transfer between the wrong accounts.

This led to a new idea to use the description as a signal. For example, in the transaction description: ONLINE TRANSFER REF #XYZ1234 TO SAVINGS XXXXXX1234 ON 11/2/11 we can see that I transferred to account X1234. I know that this account corresponds to Firefly account XYZ and can seed the algorithm to search for credits in that account.

Not all transfers include the exact account id though, especially ones that cross banks will usually show something vague. Like a transfer from one bank to another bank might just show FOOBANK ONLINE TRANSFER and that bank has multiple candidate accounts.

1
2
3
4
5
6
7


known_account_ids = {
 'X1234': 1,
 'X4561': 3,

 'FOO BANK $TRANSFER': [1, 2, 3, ...],
 # ...
}

This says if we find a transaction that says X1234, then we should look for matching transactions in account id 1. It also says that if we have transaction in account 1, we should try to find transactions that contain either X1234 or FOO BANK $TRANSFER, and if neither can be find, expand the scope to any matching transactions. So in the following example we have two transfers that all have the same amount, but must be disambiguated.

#	Description	Amount	Type	Account	Pair
1	Transfer to X1234	$100	debit	4561	4
2	Transfer	$100	credit	3	3
3	Transfer to FOO BANK $TRANSFER	$100	debit	5678	2
4	Deposit	$100	credit	1	1

In transaction #1, we’d find X1234 and look for transactions in account 1, thus finding #4 as the pair. In transaction #2, we have no account ids, but we if we look for transactions that contain either FOO BANK $TRANSFER or X4561, then we find transaction #3.

Wiring this into the algorithm is hairy, but here goes nothing. First, we identify any matching account ids defined above, along with the inverse.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


def attempt(df, base, attempt):
 if len(df[base & attempt]) > 0:
 return base & attempt
 else:
 return base

def find_transfer(row, df, time_window_days=5):
 # ...
 match_by_description = set()
 inverted_by_account_id_in_description = []
 for key, value in known_account_ids.items():
 # If we find an account id in this record's description
 # then refine the other side by the found account id(s)
 if key in row['Original Description'].upper():
 if isinstance(value, list):
 for id in value:
 match_by_description.add(id)
 else:
 match_by_description.add(value)

 # Find possible descriptions based on this row's
 # account id
 if (isinstance(value, list) and row['AccountId'] in value) or value == row['AccountId']:
 foo.append(key)
 inverted_by_account_id_in_description.append(df['Original Description'].str.contains(key, case=False))

Then we use this information to attempt different variations of the pair finder algorithm and if any of them return a result, we use that transactions as our pair.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


 # ...
 if len(match_by_description) > 0:
 opposing_filter = attempt(df, opposing_filter, df['AccountId'].isin(match_by_description))
 if len(inverted_by_account_id_in_description) > 0:
 filter_by_invert = reduce(lambda x, y: x | y, inverted_by_account_id_in_description)
        if len(not_inverted) > 0:
            filter_by_invert = filter_by_invert & reduce(lambda x, y: x & y, not_inverted)

        opposing_filter = attempt(df, opposing_filter, filter_by_invert)

 transfer_pair = df[opposing_filter]
 if not transfer_pair.empty:
 # ...

Ambiguous Matches

Even with all that, we can still end up with incorrect transfer matches. For example, I frequently transferred $1k blocks between different accounts and if that happened around the same time, it could still misclassify the transfer. The next step is to identify ambiguous matches– i.e. any time the solver finds two or more candidates and just picks one to proceed. These are likely candidates to investigate further. Let’s filter for those:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


def find_transfer(row, df, time_window_days=5):
 # ...
 transfer_pair = df[opposing_filter]
 if not transfer_pair.empty:
 df.at[row.name, 'Considered'] = True
 return opposite
 else:
 return None

def find_ambiguous(row, df):
 if df.at[row.name, 'Considered']:
 return None

 transfer_pair = find_transfer(row, df)
 if transfer_pair is None or len(transfer_pair) == 1:
 return None

 print(f"- {row['Date']} {row['Account Name']} \"{row['Description']}\"")
 for index, option in transfer_pair.iterrows():
 delta = option['Date'] - row['Date']
 print(f" {delta.days} day(s): {option['Account Name']} {option['Original Description']}")
 print("")

copy['Considered'] = False
copy[df_relevant].apply(func=find_ambiguous, axis=1, df=copy)

This gives me something like:

1
2
3


- 2012-03-01 Savings Account A: "DepositInternet transfer from Interest Checking account XXXXXXX6789"
 0 day(s) Spending Account: "WithdrawalInternet transfer to Online Savings account XXXXXXX1234"
 0 day(s) Ally Spending Account: "WithdrawalInternet transfer to Online Savings account XXXXXXX5678"

From here, we can see that I need to create an alias for 6789, X1234, X5678 in known_account_ids.

1
2
3
4
5


account_info_map = {
 'X6789': 5, # Firefly's account id
 'X1234': 6,
 'X5678': 7
}

Then rerun the transfer solver and it’ll fix those issues

Testing

How do you know if the transfers are working correctly? There’s no automatic solution for this. I built my rules based on trial and error. I looked through the data, found issues, added rules, and re-ran.

With this code, you can view the solved transfers:

1
2


transfers = transactions[df_filter_relevant].apply(func=process_record, axis=1, result_type='expand')
transfers[transfers['type'] == 'transfer']

You can also write it to a CSV and review it in a spreadsheet editor:

1

transfers[transfers['type'] == 'transfer'].to_csv('transfers.csv')

Once you’re satisfied, you can move on to the next step, which is loading into Firefly-iii in the next part of this series.

Conclusion & Next Steps

In the post, I walked through the Python code to identify and solve for transfers. The final code for this can be found on the GitHub repo here.

In the next post, I’ll show how to load all this into Firefly-iii.

My new Framework laptop

Wed, 17 Apr 2024 00:00:00 +0000

I was recently in the market for a new personal-use laptop and wanted to try out a Framework Laptop. I was intrigued by the idea of being able to replace any part that failed or even upgrade parts as I went. I also was frustrated with the direction that Windows 10 and Windows 11 was going.

They seemed more interested in advertising, tracking, sending notifications to increase my engagement of their apps, then just building an operating system that got out of my way and let me do my thing.

Here’s my brief thoughts.

I ordered the Framework 13" Laptop w/ AMD and installed Ubuntu Linux on it because it was considered well supported by Framework. The build quality felt very solid and I appreciated the ability to pick and choose what ports I had. Most laptops are switching to USB C, but I still had a number of USB A devices like my Yubikey or even slapping an HDMI port or Ethernet port in the cases that I need to do some network troubleshooting.

I encountered a few issues, but here’s how to fix them.

Issue 1 - Flickering screen

The first issue I saw was that my screen would randomly flicker white. Some others that it was a bug in the AMD drivers.

The fix was quick. All I had to do was add the the Kernel arg amdgpu.sg_display=0 to the boot config:

/etc/default/grub

1
2
3
4
5
6
7


 GRUB_DEFAULT=0
 GRUB_TIMEOUT_STYLE=hidden
 GRUB_TIMEOUT=0
 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
-GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
+GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdgpu.sg_display=0"
 GRUB_CMDLINE_LINUX=""

Then reboot.

Issue 2 - AMD Wi-Fi Card

My next issue frequently when I’d come out of standby, the Wi-Fi card would be stuck. I wouldn’t be able to get it unstuck without a reboot which got really annoying to reboot five times a day. Running sudo iwlist scan would give me No scan results. A number of articles talked about strategies, install this driver, or change that firmware. Instead, I ordered the Intel Wi-Fi 6E AX210 card and replaced it. It took me less than five minutes to replace it and it all worked better.

Importing and cleaning Mint transactions

Sun, 07 Apr 2024 00:00:00 +0000

Since Intuit announced that Mint was going away, I’ve spent several months investigating how to import my Mint data into Firefly-iii, an open source, self-hosted budgeting software. It seemed like a perfect fit. I would fully own the data and get to build whatever tooling I want on top.

However, before we can get there, we need to have cleaned and accurate data from Mint. As it turns out, Mint’s data actually had some errors in it that required me to go back years and fix them. In this post, I walk through the programmatic approach I took to identifying mistakes and fixing them.

Downloading

I previously talked about downloading data from Mint in Monarch Money and ad networks. Also download the daily balances. I also used the Monarch Money Data Exporter to download daily balance info.

Understanding the Firefly model

Importing into Firefly is tricky because it calculates the current balance of an account as the sum of all credits and debits. This is good in theory, but it requires you to have perfectly accurate transaction data going back to the account opening date. Most other tools, like Mint, Monarch, Personal Capital, all seem to track as a series of transactions and the balance is just collected from the bank and is not a function of the transactions. That’s easier to implement because it doesn’t require the full history and is robust against errors

Additionally, Firefly-iii supports explicitly modelling transfers between two accounts. In effect, the two separate credit and debit transactions are linked together. This is problematic (and you’ll see why) because no bank actually tracks the opposite transaction, nor does Mint.

Importing using the Firefly importer

Firefly has a default importer. It’s a separate Docker container that enables you to import from a csv file.

The importer has performance issues, doesn’t scale when loading large amounts of history. It would fail to load and crash when trying to do an entire year of transaction data.

Data from Mint isn’t clean and has a few duplicates here and there.

Identifying transfers is tough. It has no concept of being able to identify a transfer itself given two halves. The only way is to set remap the description as an opposing account, but descriptions aren’t consistent.

For example, these are all transaction descriptions from Mint with different formats. I know they’re on the CAPITAL ONE side and I know which account they pull from, but I had to manually remap every single month:

CAPITAL ONE AUTOPAY PYMT AuthDate 09-Oct
CAPITAL ONE AUTOPAY PYMT AuthDate 09-Sep
CAPITAL ONE AUTOPAY payment AuthDate 05-N ov
CAPITAL ONE CRCARDPMT

An ambiguous example:

Internet transfer from Interest Checking account
Internet transfer from Online Savings account

Which account was it transferred from? This simply wasn’t going to work, so I gave up using the tool.

Importing using Pandas

Pandas is a popular Python package designed for data processing and manipulation and great for working with CSV files like this. It should be more scalable and able to handle these ambiguous cases.

Loading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


import pandas as pd

transactions = pd.read_csv(f"/data/transactions.csv", parse_dates=['Date'], encoding='utf8').sort_values('Date').reset_index()
transactions['Id'] = transactions.index

# Map the Account Name column to the right Firefly account id
account_map = {
 "Checking": 2087,
 # ...
}

transactions['AccountId'] = transactions['Account Name'].map(account_map)
transactions[transactions['AccountId'].isnull()].groupby('Account Name').count()
transactions = transactions[transactions['AccountId'].notnull()]
transactions['AccountId'] = transactions['AccountId'].astype('int')

transactions['AbsoluteAmount'] = transactions[['Amount', 'Transaction Type']].apply(lambda x: x['Amount'] if x['Transaction Type'] == 'credit' else -x['Amount'], axis=1)
transactions['RunningBalance'] = transactions.sort_values('Date').groupby('Account Name')['AbsoluteAmount'].transform(pd.Series.cumsum)
print(len(transactions))

Comparing expected vs actual balances

First, we need to check to see if Mint even has the right transactions (foreshadowing). Let’s compute the total balance based on all transactions:

1
2
3
4
5
6
7
8


pd.set_option('max_colwidth', 200)
pd.set_option('display.width', 200)
pd.options.display.float_format = '{:20,.2f}'.format

balances = transactions.groupby(['Account Name', 'Transaction Type'])['Amount'].sum().to_frame().reset_index().pivot(index='Account Name', columns='Transaction Type', values='Amount')
balances['balance'] = balances['credit'] - balances['debit']

print(balances)

Next, load the balances as measured by Mint. The folder /data/mint-balances contains .csv files with the schema: Account Name,Date,Amount. I downloaded this using the Monarch Money Data Exporter before Mint shut down entirely.

1
2
3
4
5
6
7


import glob
dfs = []
for name in glob.glob('/data/mint-balances/*.csv'):
 dfs.append(pd.read_csv(name, parse_dates=['Date']))
actual_balances = pd.concat(dfs, axis=0, ignore_index=True)

current_balance = actual_balances.sort_values(by='Date', ascending=False).groupby('Account Name').first().drop(columns=['Date'])

Join to find error

1
2
3
4
5


balance_combined = pd.merge(left=current_balance.rename(columns={'Amount': 'Actual'}), right=balances.rename(columns={'balance': 'Estimated'}), left_index=True, right_index=True)

balance_combined['Error'] = balance_combined['Estimated'] - balance_combined['Actual']

print(balance_combined.sort_values('Error') 'Estimated', 'Actual', 'Error']])

Stock brokerage accounts I focused just on the cash based accounts (excluded anything with any type of security pricing such as brokerages, IRAs, etc.) and then compared the balances with the actual balances to find the balance error. In theory, if Mint were have collected data perfectly accurately, all error balances would be $0. However, it was not.

Account Name	balance	actual_balance	balance_error
A	$4,000.0	$2,000.0	$2,000.0
B	$500.00	$500.0	$0.00
C	-$1,000.0	$600.00	-$400.00

Unfortunately, errors are hard to track down because the error can come from anywhere.

Duplicates

One source is duplicated transactions. Duplicated transactions aren’t guaranteed to be bad, but could be. For example, if you pay with your phone at MTA, you’ll end up with a transaction for the same price for each ride.

1
2
3


transactions[transactions.duplicated(subset=['Date', 'Account Name', 'Amount', 'Transaction Type', 'Original Description'], keep=False)]

# [98 rows x 12 columns]

How you actually verify whether the duplicates are correct will vary. If you have access to the original statements (not likely if you’re past 7 years), then you’re going to have to use intuition. For example, I purchased something at a 0% financed rate. The payment is the same every month, so if there’s a duplicate, it’s wrong. My Capital One card allowed me to fetch any statements from the open date, my Citi and Amex cards required me to request transactions, and other cards I was out of luck.

One account, like the “Account A/(Fidelity Cash Management)”, had an error of -$43.47. This was easy because I found a duplicate transaction:

Date	Description	Type	Amount
9/25/2018	Cash Withdrawal	debit	43.47
9/25/2018	Cash Withdrawal	debit	43.47

Removing one from my .csv file brought the error for that account from -$43.47 to $0.00.

Comparing balance over time to identify errors

Once I identified which accounts showed errors, I compared the estimated balance vs the actual balance. The estimated balance is the cumulative sum of the credits and debits each day, and the actual balance is the daily balance as measured by Mint. This balance is collected when it scrapes the account so it should be accurate. Before this we compared just the ending balance, but this approach breaks it down as a time series which should help us narrow down to the exact date when an error is introduced.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


import numpy as np
from datetime import timedelta
import matplotlib.pyplot as plt

def render_bal_chart(acc_name):
 fig, ax = plt.subplots(figsize=(10, 5))
 ax.title.set_text('Difference in expected balance vs actual')
 ax.title.set_label('Date')
 ax.yaxis.set_label('$')

 actual_calc = actual_balances[actual_balances['Account Name'] == acc_name].set_index('Date')
 actual_calc['Change'] = actual_calc['Amount'].diff()
 actual = actual_calc[actual_calc['Change'] != 0]['Amount']
 estimated = transactions[transactions['Account Name'] == acc_name].groupby(['Date'])['RunningBalance'].sum()

 merge = pd.merge_asof(left=actual.rename('Actual'), right=estimated.rename('Estimated'), left_index=True, right_index=True, tolerance=timedelta(days=5), direction='backward')
 merge['Error'] = merge['Actual'] - merge['Estimated']

 ax.plot(merge['Error'])

render_bal_chart('Account Name')

This next diagram shows the difference in the daily balance calculation.

This chart is very noisy and not helpful yet. But wait, let’s zoom in! There seems to be a noticeable pattern:

As it turns out, the Mint collected account balance isn’t always right because 1) Mint doesn’t update every day and 2) pending transactions aren’t always included in the balance. To fix this, we can calculate a rolling average and drop out the error:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


import numpy as np
from datetime import timedelta
import matplotlib.pyplot as plt

def render_bal_chart(acc_name):
 fig, ax = plt.subplots(figsize=(10, 5))
 ax.title.set_text('Difference in expected balance vs actual')
 plt.xlabel('Date')
 plt.ylabel('$')

 actual_calc = actual_balances[actual_balances['Account Name'] == acc_name].set_index('Date')
 actual_calc['Change'] = actual_calc['Amount'].diff()
 actual = actual_calc[actual_calc['Change'] != 0]['Amount']
 estimated = transactions[transactions['Account Name'] == acc_name].groupby(['Date'])['RunningBalance'].sum()

 merge = pd.merge_asof(left=actual.rename('Actual'), right=estimated.rename('Estimated'), left_index=True, right_index=True, tolerance=timedelta(days=5), direction='backward')
 merge['Error'] = merge['Actual'] - merge['Estimated']

 # Denoise the signal
 rolling_mean = merge['Error'].rolling(5).mean()
 z_scores = np.abs(merge['Error'] - rolling_mean)
 df_filtered = merge[z_scores < 1]
 diff = df_filtered['Error'].diff()
 non_zero_indices = df_filtered[(diff > 1) | (diff < -1)].index

 indices = []
 # Include the previous value at each step
 for i in non_zero_indices:
 indices.append(df_filtered.index.get_loc(i) - 1)

 indices_to_include = df_filtered.index[[0, -1]].union(non_zero_indices).union(df_filtered.index[indices])
 ax.plot(df_filtered.loc[indices_to_include]['Error'], marker='.')

 for idx in indices:
 idiff = diff.index[idx]
 y = df_filtered.loc[idiff, 'Error']
 plt.text(idiff, y - 300, f"({idiff.strftime('%b %d')})", verticalalignment='bottom')

 for idx in non_zero_indices:
 idiff = diff.loc[idx]
 plt.text(idx, df_filtered.loc[idx, 'Error'], f"({idx.strftime('%b %d')}, {idiff:.2f}, {df_filtered.loc[idx, 'Error']:.2f})", verticalalignment='bottom')

render_bal_chart('Account Name')

Now I know to check between Mar 6th and April 26, 2013 for either an extra $20 credit or a missing $20 transaction. As it turns out there were two $20 credits. The same thing happened with the $1k difference in the end of 2019. This seems to be the common mistake.

This method worked pretty well for cash based accounts, but trying to apply it to a credit card broke down because every transaction goes through a pending phase for a few days, far more than the cash accounts did. Even after tweaking the denoising algorithm, it still wasn’t able to find the error points even though I could very clearly see it with my eyes.

I realized the most common failure point seems to be duplicated transactions, so can I specifically test for them? I came up with the following algorithm, first I identified all of the duplicated transactions, then it simulates removing one of the duplicates, if it reduces the error, it saves the transactions for further testing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


def render_bal_chart(acc_name: str):
 """
 Renders a chart showing the estimated balance and the correct balance for a single account.

 :param str acc_name: The account name as listed in the transactions.csv and the mint-balances.csv file
 """
 fig, (ax, ax2) = plt.subplots(2, figsize=(20, 20))
 ax.title.set_text(f'Difference in expected vs actual: {acc_name}')
 ax.xaxis.set_label('Date')

 applicable_transactions = transactions[transactions['Account Name'] == acc_name]

 actual_calc = actual_balances[actual_balances['Account Name'] == acc_name].set_index('Date')
 actual_calc['Change'] = actual_calc['Amount'].diff()
 actual = actual_calc[actual_calc['Change'] != 0]
 estimated = applicable_transactions.groupby(['Date'])['AbsoluteAmount'].sum().cumsum()

 merge = pd.merge(left=actual['Amount'].rename('Actual'), right=estimated.rename('Estimated'), left_index=True, right_index=True, how='inner', validate='one_to_one')
 merge['Error'] = merge['Actual'] - merge['Estimated']

 current_error = abs(merge['Error'].iloc[-1])
 legend_handles = []
 best_errors = []
 for index, duplicate in applicable_transactions[applicable_transactions.duplicated(subset=['Amount', 'Transaction Type', 'Date', 'Original Description'])].groupby(['Date', 'Transaction Type', 'Original Description']).agg(['count', 'sum', 'mean'])['Amount'].iterrows():
 date_key = duplicate.name[0]
 if duplicate.name[1] == 'debit':
 adj = -duplicate['mean']
 else:
 adj = duplicate['mean']
 test_s = merge['Error'].copy()
 test_s.loc[test_s.index > date_key] += adj
 new_error = abs(test_s[date_key:].mean())

 if new_error < current_error:
 best_errors.append({'error': test_s.abs().sum(), 'data': duplicate, 'adj': adj}) # Area under the curve

Once it has a list of possibly incorrect duplicates ordered by how much it reduces the error, and iteratively simulates removing multiple transactions until it no-longer improves. These transactions are the ones that have a high likelihood of being invalid. To confirm, use your memory or check your statements.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


best_errors = sorted(best_errors, key=lambda x: x['error'])
new_df = merge['Error'].copy()
current_error = new_df.abs().sum()
convergence = [current_error]
for best_error in best_errors:
 attempt = new_df.copy()
 date_key = best_error['data'].name[0]
 adj = best_error['adj']
 attempt.loc[date_key:] += adj
 new_error = attempt.abs().sum()
 if new_error < current_error:
 new_df = attempt
 current_error = new_error
 convergence.append(current_error)
 print(f"Found duplicate that when removed, reduced error: {date_key.strftime('%Y-%m-%d')}, tamount={adj},iamount={adj/duplicate['count']}")
 foo = ax.plot(new_df, label=f"Duplicate: {date_key.strftime('%Y-%m-%d')}, amount={best_error['data']['mean']}")[0]
 legend_handles.append(foo)
 ax.axvline(x=date_key, color=foo.get_color())

ax2.title.set_text('Convergence of error')
ax2.plot(convergence)

Then finally render the simulations:

1
2
3


legend_handles.append(ax.plot(merge['Error'], label='Default')[0])
ax.legend(handles=legend_handles)
ax.axhline(y=0, color='grey', dashes=[2])

Running this gets us the following. The second graph shows the error duplicates are removed (Lower is better). The first graph shows the default line along with the simulated removals. It’s a little hard to see, but at the very end, the error gets corrected back to 0 by removing the duplicates.

Note, this may have false positives or false negatives. It’s calculating based on what looks right. However, it should give you some places to look. In my case, I looked at these transactions and deleted them from the CSV because they were wrong.

Dummy Transactions

The last strategy I employ is to just add dummy transactions to correct any errors. Find the earliest statement you can get for an account and see what the starting balance is. Then compare that with your estimated balance. Whatever the difference is can be added as an initial balance transaction.

1
2
3
4


account = transactions[transactions['Account Name'] == 'General Fund'].sort_values('Date', ascending=True)

first_known = account[(account['Date'] < '2016-01-01')].iloc[0][['Date', 'RunningBalance']]
print(first_known)

To calculate the initial balance, subtract the RunningBalance from the expected balance and create a manual transaction:

1
2
3


Date 2011-11-28 00:00:00
RunningBalance 335.21
Name: 16, dtype: object

All Together

If you want to see the (messy) code all together, take a look at the GitHub repo ajacques/own-your-finances.

Next Steps & Conclusion

Now, we should have a Pandas DataFrame containing cleaned and accurate transaction data. We identified multiple strategies to identify bad transaction data In Part 2, I’ll walk through how to identify the transfer pairs and actually import it into Firefly. At some point, I’ll figure out how to link this up to pull transaction data from accounts.

If you’re in the United States, then you basically have no good options for accessing your data yourself without paying other companies (e.g. Plaid, Akoya) to screen scrape or integrate with restricted APIs. I’ll dive into this more in a later post, but this is why I’m very interested in the CFPB’s proposed rules on CFP1033. For awhile they were accepting public comments here, but that’s closed now.

Errata

2024-04-25 - Fix bug in render_bal_chart because I had an extra parameter that shouldn’t apply to other users. Added better validation.

What to expect when you're excepting Java

Fri, 08 Mar 2024 00:00:00 +0000

Birthing code is not always easy.

Enough puns. Let’s talk about Java exceptions. No matter how hard you try, your code will likely encounter an error and throw an exception (if your language supports exceptions.) It could be anything from unexpected user input to an underlying service outage. An exception will be thrown and it’s important to do something useful with it. That doesn’t mean putting try-catch blocks everywhere or trying to recover everything, in-fact I’ll argue the opposite in a few situations.

This post introduces a few common issues I’ve seen when working with Java code-bases and developers that lead to poor debuggability or other operational pains.

Practices

Stop the catch-log-throw shuffle

This is a common problem in Java code that I see. At multiple levels, developers will catch broad exceptions because there are checked exceptions just to wrap an re-throw or to log to the console.

1
2
3
4
5
6


try {
 // Do a lot of logic
} catch (Exception e) {
 log.warn("Something bad happened!", e);
 throw new RuntimeException("Failed to update.", e);
}

Then this happens at every level in the call stack and soon enough your error log is 10 screens high of call stacks and the exception class has lost all meaning.

Avoid needless wrapping of exceptions. Every wrapped exception adds more noise to understanding the problem. Each wrapped exceptions should provide some additional context or an abstracted exception type.

Take the above example:

The outer RuntimeException provides no additional context and the exception class is not specific at all. Make the error clear and direct. Is it important to know which item failed to update?
The log statement probably happens at every single catch block. The application log is probably full of useless log statements. Instead, log only at the high point in you call stack.

Logging with Log4j/Slf4j correctly

The following example breaks down a common way developers catch and log exceptions in code. Can you spot the issue? Hint: it has to do with the log format.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


import lombok.extern.slf4j.Slf4j;

@Slf4j
public class ExceptionTest {
 public static void B() {
 throw new IllegalArgumentException("Bad argument");
 }

 public static void A() {
 try {
 B();
 } catch (Exception e) {
 throw new RuntimeException("Something went wrong", e);
 }
 }

 public static void main(String[] args) {
 try {
 A();
 } catch (Exception ex) {
 // Two examples of problematic logging
 log.error("Oh no! an error happened doing {}: {}", "something", ex);
 log.error("Oh no! an error happened doing {}", ex);
 }
 }
}

Running the above code gets the following log statement. Where’s the call stack or the nested exception information?

1

15:21:08.748 [main] ERROR ExceptionTest - Oh no! an error happened doing something: java.lang.RuntimeException: Something went wrong

As it turns out, Log4j has special logic when it’s formatting the log statement. If there are n placeholders and n+1 arguments and the last value extends from Throwable, then it prints out the exception call stack and nested exceptions. If there are n placeholders and n arguments, it doesn’t matter if the last argument is a Throwable, it calls #toString() on all the objects.

The one edge case to this is if you call log.error(“msg”, throwable), then the compiler directly calls Logger#error(String,Throwable), but the end result is the same.

Instead, don’t include a log placeholder for the exception and always pass it as the last parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


import lombok.extern.slf4j.Slf4j;

@Slf4j
public class ExceptionTest {
 public static void B() {
 throw new IllegalArgumentException("Bad argument");
 }

 public static void A() {
 try {
 B();
 } catch (Exception e) {
 throw new RuntimeException("Something went wrong", e);
 }
 }

 public static void main(String[] args) {
 try {
 A();
 } catch (Exception ex) {
 log.error("Oh no! an error happened doing {}", "something", ex);
 }
 }
}

When we run the above code, we now get a much more intuitive log statement:

1
2
3
4
5
6
7
8


15:21:47.644 [main] ERROR ExceptionTest - Oh no! an error happened doing something
java.lang.RuntimeException: Something went wrong
 at ExceptionTest.A(ExceptionTest.java:13)
 at ExceptionTest.main(ExceptionTest.java:19)
Caused by: java.lang.IllegalArgumentException: Bad argument
 at ExceptionTest.B(ExceptionTest.java:6)
 at ExceptionTest.A(ExceptionTest.java:11)
 ... 1 more

If you’re using IntelliJ, you can automatically catch issues like this. It’s won’t detect all issues, but it’s a good start.

1
2
3
4
5


// This is detected
log.error("Oh no! an error happened doing {}", ex);

// This isn't detected
log.error("Oh no! an error happened doing {} {}", "something", ex);

Find it in: IntelliJ Inspections: Java | Logging | Number of placeholders does not match number of arguments in logging call.

Include Inner Exceptions

Almost always include the inner exception when throwing an exception in a catch block. Unless you’re careful to include sufficient to explain what happened, you’re more likely to throw an exception that contains not enough information. Special care should be taken when exposing exceptions outside a security boundary, such as to a caller of a service. Log the full details, but then truncate to a minimal amount of information in the response body.

For example, take this:

1
2
3
4
5


try {
 // Do some work
} catch (IllegalArgumentException e) {
 throw new WebApplicationException("Invalid argument");
}

With this, the WebApplicationException does not have an inner exception, so consumers have no context about what happened. In this specific case, it’s critical for the caller to know what argument was invalid so they can fix it.

Include an inner exception when wrapping the exception:

1
2
3
4
5


try {
 // Do some work
} catch (IllegalArgumentException e){
 throw new BadRequestException(e);
}

Don’t accidentally mistake developer mistakes for validation issues

NullPointerExceptions are commonly thrown by validation functions like Lombok’s @NonNull or Guava’s Preconditions. They’re also frequently caused by developer mistakes when you call a method on a null. Unfortunately this can lead to poor exception handling if you assume that NPEs are only thrown by your validation code.

Take the following example. NullPointerExceptions can be thrown in any line of code here. It could mean that the variable something is null and it’s possible handle it or it could mean a developer made a mistake (as I did) and tried to call a method on a null object.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


import com.google.common.base.Preconditions;

// [...]

FooBar fooBar = null;

try {
 Preconditions.checkNotNull(something.getValue(), "Validate Something");
 fooBar.callSomething();// Will throw an NPE
} catch (NullPointerException e) {
 LOG.info("something wasn't valid. It's okay though");
 // something must be null and invalid
}

Extract out exception message construction

If you create and throw custom exceptions, you might find yourself writing code like this:

1
2
3
4
5
6
7
8


class MyCustomException extends RuntimeException {
 public MyCustomException(String message, Throwable cause){
 super(message, cause);
 }
}

// ...
throw new MyCustomException(String.format("Something %s happened when doing %s while also doing %s", x, y, z));

If you throw this exception multiple times, then you may end up copying and pasting the String.format code in multiple places. Instead just move the message construction onto the exception class itself:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


class MyCustomException extends RuntimeException {
 public MyCustomException(String x, y, z, Throwable cause) {
 super(makeMessage(x, y, z), cause);
 }

 private static String makeMessage(String x, String y, String z){
 return String.format("Something %s happened when doing %s while also doing %s", x, y, z);
 }

}

// ...
throw new MyCustomException(x, y, z);

Structured exceptions

Pretty much every exception I see thrown converts all the problem details into a single giant string message. Strings are terrible at encoding information. Sure a human can read it, but often times code needs to analyze it. Say you’ve got a library that throws a throttling exception when the client calls it too frequently:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


void performSomething() {
 if (numCalls > 100) {
 throw new ThrottledException(String.format("Request throttled. Client hit %s calls in %s seconds. Please try again in %s seconds.", ...),
 }
}

try {
 performSomething();
} catch (ThrottledException e) {
 int pos = e.getMessage().indexOf('try again in' )
 int seconds = e.getMessage().substring(pos + 'try again in '.length);
}

As a caller, I might want to know when I actually can try again. Since this is currently in a string, I’d have to string parse this exception message to find out. That’s horrifying.

Instead, have your exception object property getters for these different facts:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


class ThrottledException extends RuntimeException {
 private final int numCalls;
 private final int numSeconds;
 private final Instant resetsAt;

 ThrottledException(int numCalls, int numSeconds, Instant resetsAt) {
 super(String.format("Request throttled. Client hit %s calls in %s seconds. Please try again at %s.", numCalls, numSeconds, resetsAt));
 this.numCalls = numCalls;
 this.numSeconds = numSeconds;
 this.resetsAt = resetsAt;
 }

 /**
 * Gets when the caller can resume performing work.
 **/
 public Instant getResetsAt() {
 return resetsAt;
 }

 // ...
}

void performSomething() {
 if (numCalls > 100) {
 throw new ThrottledException(numCalls, numSeconds, Instant.now().plusHours(1));
 }
}

try {
 performSomething();
} catch (ThrottledException e) {
 e.getResetsAt(); // Now I know when to check again
}

RFC9457 is a good resource on how to apply this style of error structuring to an HTTP endpoint.

Checked vs Unchecked Exception

Checked exceptions are exceptions which are checked by the compiler, if they are being explicity thrown or caught. The class Exception and any subclasses that are not also subclasses of RuntimeException are checked exceptions. Checked exceptions need to be declared in a method or constructor’s throws clause if they can be thrown by the execution of the method or constructor and propagate outside the method or constructor boundary.

Examples of checked exceptions that we might have seen are IOException, JSONParseException.

Every exception which is subclass of the RuntimeException class is an unchecked exception and not checked by the compiler. A common example of this is the NullPointerException.

Which type of exception should we use when creating our own exceptions?

Almost always the recommendation is to create an unchecked exception.

Checked exceptions, if not modeled correctly, adds unnecessary burden on the developers. If you throw a checked exception in your code and it can only be appropriately handled, say 3 - 4 layers up, it has to be added in each method’s declaration. Imagine adding/removing a checked exception in the code which would require layers of code to be refactored. Checked exceptions also do not add any value in our service interfaces as the actual clients will never catch it.

The worst offender for useless checked exception is Charset#ofName which throws a checked IllegalCharsetNameException. Every time I’ve used it, I’ve always passed the static string UTF-8 and if it’s missing, I’m screwed.

Imagine, every time you have to deserialize or serialize a file you end up with this the following code and this untestable catch block that reduces my code coverage.

1
2
3
4
5


try {
 Charset.ofName('UTF-8');
} catch (IllegalCharsetNameException e) {
 throw new RuntimeException("When would this ever happen?", e)
}

References

Monarch Money and ad networks

Wed, 14 Feb 2024 00:00:00 +0000

In late 2023, Intuit announced that Mint was going to be shutting down and migrating everybody to Credit Karma. I could try out Credit Karma, but maybe it’s time to explore alternatives. Since that announcement came out, I launched a massive time sink to try and find a new option I liked.

Intro

Different people have different goals for a finance tracking app. My requirements are not so much about following a strict budget, but more focused on transaction tracking, tracking stocks and other assets, and future planning. I looked around and found a few.

The first one I tried is Monarch Money.

Monarch Money was interesting. It was $99/yr, but with a MINT50 discount it was only $50/yr. I don’t mind paying for services. Especially ones that add value. I tried it out. It had a mechanism to import data from Mint.

Importing from Mint

This is broken down into the transactions and the balance history. This was tedious, but largely worked. I first discovered that Mint truncated the transaction.csv history to 10,000 records and lost data.

To fix this, export each section independently until you have under 10k in each export and join them together into a single CSV.

The next problem is that Monarch Money requires all accounts to be created before importing which meant I couldn’t import accounts that were closed. This is not a huge deal, but I had 70+ accounts at one point or another.

This was fine, took effort to work. I had to create all the accounts first, the import transactions. It also struggled with CDs which got opened and closed.

Ad networks?

But then I noticed something interesting in NoScript. It loaded tons of assets from other domains. Why is a financial institution that has access to all my financial data loading data from tiktok.com?!

1
2
3
4
5
6
7
8


<script src="/analytics.js"></script>
<script src="/reddit.js"></script>
<script src="/spotify.js"></script>
<script src="/tiktok.js"></script>
<script src="/clarity.js"></script>
<script src="/userleap.js" userleap_id="jhOvgs1si6">

<script type="text/javascript" async="" src="https://analytics.tiktok.com/i18n/pixel/events.js?sdkid=CAG18GJC77U2NHFFNB3G&lib=ttq">

I looked at /reddit.js, and at the time it showed this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


!(function (w, d) {
 if (!w.rdt) {
 var p = (w.rdt = function () {
 p.sendEvent ? p.sendEvent.apply(p, arguments) : p.callQueue.push(arguments);
 });
 p.callQueue = [];
 var t = d.createElement('script');
 (t.src = 'https://www.redditstatic.com/ads/pixel.js'), (t.async = !0);
 var s = d.getElementsByTagName('script')[0];
 s.parentNode.insertBefore(t, s);
 }
})(window, document);
rdt('init', 't2_5u6sm01h');
rdt('track', 'PageVisit');

Now this looks like a tracking pixel which is used to ad conversion tracking. It’s intended to track if a user were to click on an ad, then sign-up, it would count towards marketing as a success. This means it does not appear to be sending any net worth information to Reddit to build up a profile (at least in this code), it is trying to see how many people are clicking ads on Reddit, Spotify, TikTok and signing-up. However, since it is loaded as a script tag, nothing stops those companies from injecting code in your browser.

They even call this out on their help page here. However, there’s no reason that click attribution scripts should be loaded while signed-in to my account. At the very least they should only be loaded on user sign-up pages prior to creating an account. Even better is not to load it at all, but that’s always a battle with the marketing departments.

To me, it shows poor security practices. Not a great look when dealing with critical financial software. I’m still signed up for the demo, but I’ll continue to update my opinion as I use it.

Migrating from Google Location History to OwnTracks

Mon, 01 Jan 2024 00:00:00 +0000

I’ve been slowly reducing the amount of data shared with Google. I’ve been using Google Location History since 2013. I found it really useful just because I could figure out what restaurant I went to when I was traveling or any number of things.

I found OwnTracks which was an open-source location history storage solution. It’s not nearly as polished as Google Maps where it natively integrates your location history, but step one is owning my data, step 2 can be better UIs.

In this post, I’m going to walk through exactly how to get data from Google Location History and import it into OwnTracks

Setup OwnTracks

Feel free to follow the guides here to setup the OwnTracks recorder and UI.

Exporting

Google is moving their location history from server side to device side news to be ensure that they don’t have access to your location data, but as of now (August 2024) I’ve not been able to export my data on my phone.

Android

It looks like Google is trying to add support for exporting

Settings
Location
Location Services
Export timeline date
Authenticate
Pick a folder and then save it

Then if you’re lucky, the save will succeed and you can upload it to a computer. Mine, unfortunately, which failed with an error and it created a partially generated file. There appears to be an application bug where Google Maps isn’t able to handle certain timeline entries.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


Shushing crash.
FATAL EXCEPTION: lowpool[2]
Process: com.google.android.gms, PID: 28769
java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.Class java.lang.Object.getClass()' on a null object reference
 at erfs.A(:com.google.android.gms@243333039@24.33.33 (190408-666381490):1)
 at dgix.f(:com.google.android.gms@243333039@24.33.33 (190408-666381490):672)
 at bopo.fx(:com.google.android.gms@243333039@24.33.33 (190408-666381490):1)
 at boqa.run(:com.google.android.gms@243333039@24.33.33 (190408-666381490):66)
 at epiu.run(:com.google.android.gms@243333039@24.33.33 (190408-666381490):21)
 at amrj.c(:com.google.android.gms@243333039@24.33.33 (190408-666381490):50)
 at amrj.run(:com.google.android.gms@243333039@24.33.33 (190408-666381490):76)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
 at amwv.run(:com.google.android.gms@243333039@24.33.33 (190408-666381490):8)
 at java.lang.Thread.run(Thread.java:1012)
 Suppressed: epjf:
 at tk_trace.314-ExportOperation(Unknown Source:0)
 at tk_trace.semanticlocationhistory-SemanticLocationHistoryZeroPartyClientChimeraService-ISemanticLocationHistoryZeroPartyService_7(Unknown Source:0)

iOS

Google Maps
Your Timeline (three dots, top right corner)
Location and privacy settings
Export Timeline Data

Google Takeout

If you haven’t migrated to on-device location tracking, then you can use the Google Takeout method:

https://takeout.google.com/
Deselect all, select Google location History
Export once, file size: 4GB
Wait for it to export, then download and extract the .zip file

Importing into OwnTracks

I first tried to follow this guide which uses this importer in the OwnTracks/recorder repository. My exported JSON file was 2GB, and it took hours to import and the MQTT importer seemed to have dropped a few records here and there.

Instead, let’s try to import it directly and bypass MQTT. I opened up OwnTrack’s /store folder and checked out the file format:

1
2
3
4


ls /[...]/rec/user/phone# head 2023-01.rec
2023-01-01T00:00:52Z * {"_type":"location","tid":"fl","tst":1672531252,"lat":47.12345,"lon":-122.12345,"acc":13,"alt":123,"vac":4}
2023-01-01T00:01:52Z * {"_type":"location","tid":"fl","tst":1672531312,"lat":47.12345,"lon":-122.12345,"acc":13,"alt":123,"vac":4}
[...]

The file was plain-text and appeared easy to generate. I created a Python function that loaded data from /data/Records.json then generated the text files to /data/location/yyyy-mm-.rec.

I created a Git repo with my tools in a much cleaner format.

Install it using:

git clone https://github.com/ajacques/google-location-history-tools.git
uv sync

New Format

If you’ve exported your location history from your phone using the new Android or iOS export mechanism (not Google Takeout), then use this code to load the data. You’ll still need to run the code in Processing.

1

uv run main.py Timeline.json --gmaps --tracker-id [two digit owntracks tracker id]

Old Takeout Format

If you’ve exported your data from Google Takeout, prior to Google deprecating that mechanism, then use this code to load the data. You still need to run the code in the next section.

1

uv run main.py Records.json --takeout --tracker-id [two digit owntracks tracker id]

Processing

Once you have your data from the above code, you need to run the following to process and save the data. Make sure to set the –tracker-id to something meaningful.

After it generates the data, find your OwnTracks data directory. The copy the contents of the output/ folder into /{owntracks_store_dir}/store/{user}/{device} folder. If you replace any existing data you will lose data recorded from Opentracks itself, so I would recommend downloading the Google Location History data first, converting, importing, then start collecting data afterwards.

Removing extra devices

When I loaded this, the history was crazy showing me moving in and out of a city instantly:

Turns out, multiple devices on my accounts were reporting locations so it appeared that I kept moving between two cities. Easy enough to exclude the device.

The following script produces a chart that shows when devices are reporting their location:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


df_gps.sort_values(by='timestamp', inplace=True)
devices = df_gps['deviceTag'].unique()

plt.figure(figsize=(10, 6))
colors = plt.cm.viridis_r([i / len(devices) for i in range(len(devices))])

for i, device in enumerate(devices):
 device_data = df_gps[df_gps['deviceTag'] == device]
 first_year = device_data['timestamp'].dt.year.min()
 last_year = device_data['timestamp'].dt.year.max()
 middle_year = (first_year + last_year) / 2

 plt.hlines(y=i, xmin=first_year, xmax=last_year, color=colors[i], linewidth=4)
 plt.text(middle_year, i + 0.25, i, verticalalignment='center', fontsize=8)

plt.xlabel('Year')
plt.ylabel('Device')
plt.title('Device Reporting Years')
plt.grid(True)

plt.show()

With this I can identify which device is reporting in parallel. In my case, it’s the 10th device that’s reporting in parallel with my phone. Note that I’ve replaced the device ids, yours will be a long random number:

From there, I filtered out this device from the location history

1
2
3
4
5
6
7
8
9


+excluded_devices = [1234510]

 owntracks = df_gps.rename(columns={'latitudeE7': 'lat', 'longitudeE7': 'lon', 'accuracy': 'acc', 'altitude': 'alt', 'verticalAccuracy': 'vac'})
 owntracks['tst'] = (owntracks['timestamp'].astype(int) / 10**9)
+matched = owntracks['deviceTag'].isin(excluded_devices)
+if len(owntracks[matched]) == 0:
+ print("Didn't find matching deviceTags. Double check your excluded_devices")
+ return
+owntracks = owntracks[~matched]

And the history looks a lot better:

Adding new data

Now, once you have your historical data, what do you do for new data moving forward? There’s a lot of options depending on personal preference including:

Using the OwnTracks mobile apps
Copying location from Home Assistant like here. This is the method that I use.

Conclusion

OwnTracks is an open-source system to store your location history. You can import existing history from Google Maps, then report new data into OwnTracks and have a localized copy of your location history.

Updates

2024-05-04 - Corrected a few bugs in the Python code
2024-07-15 - Automatically create output folder in Python
2024-08-02 - Updated the timestamp parsing logic again
2024-08-28 - Updated downloading section to include Android and iOS examples (thanks to reader contribution)
2024-11-24 - Clarified sections and added more validation around excluded devices
2026-03-10 - Added link to GitHub repo

Content-Security-Policy for Home Assistant

Sun, 31 Dec 2023 00:00:00 +0000

Content-Security-Policy is a security feature (MDN Web Docs) in modern web browsers that restricts the kind of content that helps to protect against certain types of attacks, such as Cross-Site Scripting (XSS) attacks. Since my Home Assistant has significant access to my home network and is reasonably well-known, I wanted to take some steps to protect against malicious actors using XSS or other injection attacks to taking over my network. In addition, there have been a few different CVEs (HA Security Disclosures) in Home Assistant that allowed for XSS),

In this blog post, I walk through the steps I took to design a CSP header and fix bugs I found along the way.

CSP won’t protect against all attacks, but it’s one tool as part of a layered security system to protect your system. CSP works by explicitly identifying all sources of JavaScript, CSS, XHR requests, etc. and explicitly listing them in an HTTP response header sent to the browser.

My Stack

I run Home Assistant inside Kubernetes which is fronted by ingress-nginx. This is important because I attach the CSP header into the response using ingress-nginx. Home Assistant itself does not expose a config value to set this header in the http integration. Using Kubernetes isn’t required,

Configure ingress-nginx

First, we need to update ingress-nginx to allow us to define headers by updating the global-allowed-response-headers setting in the ConfigMap.

If you’re using Helm, the setting is:

1
2
3


controller:
 config:
 global-allowed-response-headers: 'Content-Security-Policy'

A broken header

If you’re using Kubernetes like me, then go to the HA Ingress and start setting the header like below. Right now, the headers are defaults, but we’ll fill those in.

1
2
3
4
5
6
7


apiVersion: v1
data:
 Content-Security-Policy: default-src 'self'
kind: ConfigMap
metadata:
 name: hass-ingress-headers-full
 namespace: smarthome

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
 annotations:
 # AuthN/Z handled by HA itself
 nginx.ingress.kubernetes.io/enable-global-auth: "false"
+ nginx.ingress.kubernetes.io/custom-headers: smarthome/hass-ingress-headers-full
 name: homeassistant
 namespace: smarthome
 spec:
 ingressClassName: external-nginx
 rules:
 - host: ha.example.com
 http:
 paths:
 - backend:
 service:
 name: homeassistant-headless
 port:
 number: 8123
 path: /
 pathType: Prefix
 tls:
 - hosts:
 - ha.example.com

A vanilla nginx config will look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 server {
 listen 80;
 server_name ha.example.com;

+ add_header Content-Security-Policy "default-src 'self'";

 location / {
 proxy_pass foo:8123;
 }
 }

Traefik can use this.

A basic header

To start, I opened up my browser dev tools, reloaded the page, navigated around, then noted down all the different URLs that were loaded. Then I went to a CSP Generator. Most things load from the same HA endpoint, but things like maps and HCAS will load from external origins which must be explicitly permitted.

The header will look something like this:

default-src/Default Source - Set this to 'self' as a fallback grant if a request doesn’t match another directive.
script-src/Script Source - Set this to 'self' 'unsafe-inline' 'unsafe-eval'. This directive is the most important because it scripts are what make XSS attacks useful (more on the risks below.)
style-src/Style Source - Set this to 'self' 'unsafe-inline'
image-src/Image Source - Set this to 'self' data: https://basemaps.cartocdn.com https://*.basemaps.cartocdn.com https://brands.home-assistant.io https://raw.githubusercontent.com. cartocdn.com is used for Home Assistant’s maps, brands.home-assistant.io is used to load the integration icons, and raw.githubusercontent.com is used in HACS to view images for repositories. Make sure to add any other URLs that you’ve identified in the dev tools For example, I use the hass-tryfi integration, so I needed to add https://barkinglabs-media.s3.amazonaws.com to this header.
connect-src/Connnect Source - Set this to 'self' data: https://brands.home-assistant.io https://raw.githubusercontent.com. Interestingly, since HA uses a Service Worker the initial request to populate the Service Worker cache is considered a connect, so if an image isn’t loading, try adding it to the connect-src too.
frame-ancestors/Frame Ancestors - Set this to 'none'. Home Assistant shouldn’t be embedded in an iframe. Replaces the X-Frame-Options header, see CVE-2023-41897 and GHSA-935v-rmg9-44mw.
report-uri/Report URI - Optional. I use the Sentry integration and collect CSP reports to investigate issues using this. Example: https://sentry.example.com/api/12345/security/?sentry_key=abcdef

Add the CSP header to the request router config, reload it, then refresh the browser page with the dev tools open. If anything does not load correctly, update the CSP header as needed. Note, that if you using HACS to load custom integrations, the repository pages will be intentionally partially broken. HACS shows the README.md from the GitHub repository, and often times developers include images from other origins, such as img.shields.io as “shield icons” that show things like below:

I intentionally didn’t add this for privacy reasons and they don’t add a lot of value. However, if you want to see them, just add https://img.shields.io to your image-src.

Analysis of my header

Home Assistant is intentionally quite flexible in what JS code it can run. Since you can load arbitrary cards through HACS which bring in their own HTML, CSS, JS, and developers sometimes use inline <script>foo();</script> tags, I had to enable unsafe-inline. This is a common source of drive-by XSS attacks and would still leave me susceptible to things like CVE-2017-16782 and CVE-2023-41896 since you could still inject JavaScript through incorrectly escaped content. Unescaped content isn’t the only source of XSS attacks though.

The next layer of defense is what I set for image-src, style-src, and connect-src which are the mechanisms to then exfiltrate data to a malicious actor. Any origin listed there should be considered relevant trusted. Don’t add * or anything.

Problems with iframes in Firefox

I’m using the weather-radar-card to show precipitation clouds. This required adding https://tilecache.rainviewer.com to my image-src directive. However after setting the CSP header, it would only show the first time I loaded the page, but not if it was in the cache. I had to hard refresh (shift-F5) to get it to load. Otherwise, it’d show an empty white box, like below:

It took me awhile to figure this out, because it only seemed to happen in Firefox, but looking at the HTML DOM, it looked like this:

1

<iframe srcdoc="<html><script src="/local/community/weather-radar-card/leaflet.js"></script>

According to the spec, iframe srcdocs are supposed to inherit the parent’s Origin and thus use the same header we set above, but for some reason, it was enforcing the header, but not using the same Origin. Even though we set 'self', it was ignoring that and thought it was different. To fix this, I added ha.example.com to my script-src and image-src.

After that, the card started working:

Just for fun (this is optional), I added a second restricted header just for the login page to reduce my exposure even more.

1
2
3
4
5


Content-Security-Policy:
 default-src 'self' data:;
 script-src 'self' 'unsafe-inline';
 style-src 'self' 'unsafe-inline';
 frame-ancestors 'none'

All Together

Now, putting it all together, my Content-Security-Policy looks like:

1
2
3
4
5
6
7


Content-Security-Policy:
 default-src 'self' data:;
 script-src 'self' 'unsafe-inline' 'unsafe-eval' https://ha.example.com;
 style-src 'self' 'unsafe-inline';
 img-src 'self' data: https://ha.example.com https://barkinglabs-media.s3.amazonaws.com https://basemaps.cartocdn.com https://*.basemaps.cartocdn.com https://brands.home-assistant.io https://tilecache.rainviewer.com https://raw.githubusercontent.com;
 connect-src 'self' https://brands.home-assistant.io https://raw.githubusercontent.com;
 frame-ancestors 'none'

And inserting that into my K8s Ingress:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


apiVersion: v1
data:
 Content-Security-Policy: >-
 default-src 'self' data:; script-src
 'self' 'unsafe-inline' 'unsafe-eval' https://ha.example.com; style-src 'self' 'unsafe-inline';
 img-src 'self' data: https://ha.example.com https://barkinglabs-media.s3.amazonaws.com
 https://basemaps.cartocdn.com https://*.basemaps.cartocdn.com
 https://brands.home-assistant.io https://tilecache.rainviewer.com
 https://raw.githubusercontent.com; connect-src 'self'
 https://brands.home-assistant.io https://raw.githubusercontent.com;
 frame-ancestors 'none';
kind: ConfigMap
metadata:
 name: hass-ingress-headers-full
 namespace: smarthome
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
 # AuthN/Z handled by HA itself
 nginx.ingress.kubernetes.io/enable-global-auth: "false"
 nginx.ingress.kubernetes.io/custom-headers: smarthome/hass-ingress-headers-full
 name: homeassistant
 namespace: smarthome
spec:
 ingressClassName: external-nginx
 rules:
 - host: ha.example.com
 http:
 paths:
 - backend:
 service:
 name: homeassistant-headless
 port:
 number: 8123
 path: /
 pathType: Prefix
 tls:
 - hosts:
 - ha.example.com
---
apiVersion: v1
data:
 Content-Security-Policy: >-
 default-src 'self' data:; script-src
 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline';
kind: ConfigMap
metadata:
 name: hass-ingress-headers-login
 namespace: smarthome
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/enable-global-auth: 'false'
 nginx.ingress.kubernetes.io/custom-headers: smarthome/hass-ingress-headers-login
 name: homeassistant-login
 namespace: smarthome
spec:
 ingressClassName: external-nginx
 rules:
 - host: ha.example.com
 http:
 paths:
 - backend:
 service:
 name: homeassistant-headless
 port:
 number: 8123
 path: /auth/
 pathType: Prefix
 tls:
 - hosts:
 - ha.example.com
 secretName: home-wildcard

Conclusion

Content-Security-Policy is a useful security feature that mitigates many (but not all) types of XSS attacks in the browser. It can be tricky to set in Home Assistant because HA uses a variety of different features like service workers, iframes, and allows arbitrary JS to be loaded by design through custom cards. However, this makes it more important to add a CSP header to protect as much as possible.

Updates

2025-04-02 - ingress-nginx now considers the snippets to be a high risk annotation and it’s not enabled by default. I’ve updated the examples to use the new method

Auto disable Kubernetes' service LB NodePorts

Sun, 08 Oct 2023 00:00:00 +0000

In a previous post, I noticed that all my Kubernetes services with type=LoadBalancer were exposing some internal services as NodePorts which meant that I might be exposing internal services to the Internet at high ports. I was running Kubernetes directly on my dedicated servers and not behind a load balancer. Kubernetes expected everybody to sit behind a LB which often times required a NodePort.

The solution was to set the Service spec.allocateLoadBalancerNodePorts value to false when the service is created. This works if I can set it while I create the Service, however Helm based templates often wouldn’t allow me to set this and once it was set to true and the node port was allocated it was difficult to deallocate the NodePort.

In this post, I walk through using a Kubernetes mutating webhook to automatically set the value for all Services.

What are webhooks?

The Kubernetes admission webhooks are a Kubernetes feature that enables you to process any change made in the cluster. There are two flavors:

Validating - Checks a pending resource change and returns whether it should be denied or accepted
Mutating - Checks a pending resource change and possibly changes the values in that resource.

Webhooks get registered as a Kubernetes resource, then are called automatically.

Options

There’s a few different options:

Manually create a webhook and implement the API
OpenPolicyAgent Gatekeeper
Kyverno
KubeWarden

I first experimented with OPA Gatekeeper, but found the authoring and policy registration process to be complicated. KubeWarden was another option, but overly complex and I just needed some basic policies. Kyverno looked perfect. I could easily define policies using just YAML. Far simpler than OPA’s WebAssembly project. Thus I selected Kyverno.

Deploying Kyverno

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# values.yaml
# Cleanups are currently considered alpha. Minimize the deployment size
cleanupController:
 enabled: false
reportsController:
 enabled: false
features:
 admissionReports:
 enabled: false
 # This next feature is optional
 # Set it to true to allow fail-open if Kyverno is offline
 # In my home lab, I'm using this for house-keeping, not security
 # So I want to be able to gracefully handle errors.
 # https://kyverno.io/docs/installation/#security-vs-operability
 forceFailurePolicyIgnore:
 enabled: true
 policyReports:
 enabled: false

1
2


helm repo add
helm install --create-namespace -n kyverno kyverno/kyverno kyverno -f values.yaml

Authoring the Policy

I defined the following policy and uploaded it to my cluster. Anytime a LoadBalancer Service is created or updated, the following policy will apply and disable NodePorts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: disable-lb-node-port
spec:
 mutateExistingOnPolicyUpdate: false
 rules:
 - match:
 resources:
 kinds:
 - Service
 preconditions:
 all:
 - key: "{{ request.object.spec.type }}"
 operator: Equals
 value: LoadBalancer
 mutate:
 patchStrategicMerge:
 spec:
 allocateLoadBalancerNodePorts: false
 name: mutate-loadbalancer-service
 validationFailureAction: Enforce

Existing Resources

Any service created prior to the policy won’t be updated automatically. Even if we toggle the allocateLoadBalancerNodePorts to false, and existing nodePorts will remain allocated and accessible to the Internet.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


apiVersion: v1
kind: Service
metadata:
 name: postgres-lb
 namespace: datastore
spec:
 allocateLoadBalancerNodePorts: true
 clusterIP: 10.43.23.122
 clusterIPs:
 - 10.43.23.122
 externalTrafficPolicy: Local
 healthCheckNodePort: 32439
 internalTrafficPolicy: Cluster
 ipFamilies:
 - IPv4
 ipFamilyPolicy: SingleStack
 ports:
 - name: sql
 port: 5432
 nodePort: 30313
 protocol: TCP
 targetPort: 5432
 selector:
 cnpg.io/cluster: postgres
 sessionAffinity: None
 type: LoadBalancer

The only way to fix this is to remove the nodePort field and rename each port name temporarily to get Kubernetes to clear it out.

Unfortunately as of Kubernetes 1.26 it’s still not possible to disable the healthCheckNodePort if you’re using externalTrafficPolicy=Local

Conclusion

Using Kyverno, I showed how to automatically disable unintended node ports created by Load Balancers.

Improving bad on-call with the Snowball Effect

Sun, 25 Jun 2023 00:00:00 +0000

I’ve worked on several different teams over the past 8 years I’ve worked at Amazon. Each one of them had on-call in which the engineers were on-call to keep the system running 24/7 for a week. If something broke at 2am, they’d get paged to fix it.

Now, Amazon’s a big company. On-call varied quite a bit. Some teams had more ops load, others had barely any. I had my fair share of weeks with lots of tickets, but usually I sought out teams where it was more manageable. However, those engineers frequently struggled to get anywhere, playing a bit of on-call hot potato with the next on-call. Sadly, Amazon largely did not leverage SREs or dedicated support groups except for the most critical systems. I do wish they would have leveraged them.

Here’s my strategy that I employed when I joined a new team.

The Situation

On-call was generally structured as a week long duty that rotated through engineers on the team. On the on duty week, the on-call engineer would be responsible for keeping the system operational and responding to any emergent issues. Tickets were cut based on impact to the business and customers. Sev2s were cut any time there was a customer impacting or soon to be impacting issue, though sometimes they were cut too aggressively and the engineer would either downgrade it or it would be fixed after responding.

The ticket queues would often times have a lot of tickets in it just waiting for a response and nobody would respond because they didn’t have time. Things just stayed bad and engineers would complain about OE load.

Then during the weekly OE hand-off meeting, the previous on-call would go through the list of problems, scroll through dashboards, and give out action items. It was generally a state of despair.

Let’s walk through the strategy I employed to improve my teams.

The Strategy

Accomplishments

To instill ownership into the team, I introduced the goal that every on-call would accomplish one thing to make the next on-call’s life easier. Each on-call was to present their accomplishment during the hand-off. It could be big like building a new ops tool, it could be small like fixing an alarm that was too noisy, but I clearly drew the line between things that were just work as normal vs actually innovative.

Examples of things that wouldn’t qualify, but still important to do:

Resolving a ticket
Executing a manual deployment
Responding to a customer request

Examples of good accomplishments:

Fixing an alarm that was too noisy
Taking steps towards CI/CD by introducing better pipeline approval workflows
Spending time to root cause a reoccurring issue and proposing a permanent fix

Sometimes, engineers claimed they didn’t have time to do this because there were too many issues. This showed good ownership, they wanted to fix all the problems, but that just doesn’t scale. Closing ticket rarely reduces the number of tickets, and eventually as your service scales, you just get more and more tickets. I’ve been there, seen that, but I aligned with the Team Manager and team that it was okay to fix one fewer issue as long as you instead improved one thing.

At first it was tough for on-calls to get this mindset, but week by week, they’d slowly fix things and bit-by-bit common pain-points would be reduced further freeing up time.

Reduce Aggressive Paging

I signed up to receive every an email alert every time an engineer got paged and reviewed each one. I questioned: was there really a problem here? Did you really need to get paged at 2 o’ clock in the morning? Did you do anything? Could it have waited until 9am? How do we prevent it from happening again?

Sometimes the team would say “but this alarm could tell us something’s wrong.” How many times has it caught an actual issue? Are there more direct measurements of a problem. For example, an alarm on number of 4xx response codes on an HTTP server sounds like a good thing to measure, but there’s all kinds of challenges. Do you use a percentage or a raw count (obviously percentages), but what happens if you have a low traffic API that gets a few transactions per second and somebody refreshes on a 404 page? Boom engineer paged. Even if an alarm could raise a problem, if it’s not directly measuring a problem and in the past it’s largely been annoying, it’s better to be pragmatic and just change thresholds or even disable alarms until you’re in a better state.

Use any situation where an engineer got paged to aggressively change thresholds or remove alarms until only high confidence problems are waking engineers up at 2 in the morning.

No doom-scrolling in the OE hand-off

Poorly structured OE hand-off meetings often devolve into scrolling through dashboards pointing to things and saying “oh what’s that spike?” or scrolling through issues that need to be fixed. This quickly wastes the time of the entire team who, especially on conference calls, check out and start doing other things. Everybody knows there are problems.

Instead, the hand-off meetings should focus on what are the key problems that the on-call is facing and what steps should the next on-call take. At the start on their own time, the on-call should review dashboards and the items from the previous rotation and identify one or two goals they’ll get done that week.

Conclusion

This is not a comprehensive list of everything that will immediately improve operations, however by taking a few steps here and there, your team will eventually get better. It’s like a snowball, a few small things here, then bigger and bigger problems, eventually you’ve got a snowman.

Technical Diagrams - Stop using cloud logos

Fri, 24 Feb 2023 00:00:00 +0000

Quick, what is this diagram trying to show?

An architecture diagram using AWS service icons to describe services

I hope you know your AWS icons. There’s over 200 services and I have to guess frequently when playing the AWS Logo Quiz. While this diagram could easily add some descriptive labels to help, the icons assume developers can remember what the icon means. Some color-blind people may even struggle to see the difference in coloring that AWS uses for different types of services. These icons become visually cluttered and distract the viewer from what matters–your system.

Your design documents aren’t supposed to be AWS marketing material, they’re supposed to be refined, intuitive, and show how the parts fit together, not just show you can glue together these different services.

Compare that with the below diagram:

Here I use simple boxes and arrows to focus the viewer on the key parts that matter on each part. Service and component names become the primary focus with the service being a secondary detail on the label.

I also use colors and simple shading to signify which components are new, changing, or being removed in my projects. This helps to convey relative impact of projects to other engineers. Avoid using too many similar colors to ensure that color blind coworkers are still able to understand your diagrams.

Local Energy Monitoring using the Emporia Vue 2

Wed, 22 Feb 2023 00:00:00 +0000

I’ve previously explored the world of home energy monitoring systems and in the past arrived at using the Brultech GreenEye Monitor for a project in a friend’s house. It had the advantage of being local out-of-the-box and had a wide range of compact CTs that made fitting the electronics in the breaker box a lot easier, but it had one flaw that made it not suitable for my condo. It had to be mounted outside the breaker box with wires running into the box. I had no space in my condo, so I instead explored other options.

I came across the Emporia Vue2 and identified that it was running a standard ESP32 device and was easy to reflash with custom ESPHome firmware. ESPHome is an open-source framework for creating firmware to collect data from a variety of different sensors and publish it to MQTT/Home Assistant. This sounded perfect, so I ordered a Vue2 and here’s how I made it work.

Gear Used

Emporia Vue2 - Obviously
13 Emporia 50A CT Clamps
SparkFun Serial Basic Breakout CH340G - A USB to UART board to write firmware
Pogo Pin Probe Clip - 6 Pins with 2.54mm / 0.1" Pitch - Used to interface with the ESP32 without soldering

Installation

I found the emporia-vue-local GitHub project here emporia-vue-local/esphome and followed the guide here. I had some issues trying to wire up to the DTR and CTS pins, so I instead connected IO0 and CTS to GND at boot, then let EN go high when I was ready to program.

I tried using the ESPHome Web UI to program the device, but it never worked correctly, so instead I used ESPTool on my laptop (Installation Guide).

First I made a backup of my existing firmware:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


python3 -m esptool read_flash 0 0x400000 flash_contents.bin
esptool.py v4.4
Found 3 serial ports
Serial port /dev/cu.usbserial-130
Connecting...
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting...
Detecting chip type... ESP32
Chip is ESP32-D0WD (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: de:ad:be:ef:ca:fe
Stub is already running. No upload is necessary.
4194304 (100 %)
4194304 (100 %)
Read 4194304 bytes at 0x00000000 in 379.5 seconds (88.4 kbit/s)...
Hard resetting via RTS pin...

Then I went to my ESPHome dashboard and created a new configuration. I started with the reference ESPHome, but made a few changes. Specifically:

I updated the CTs to match my phases and circuits. You will need to do the same.

More importantly, I restructured how the sensors were configured to improve accuracy and reduce useless events. More information is found in the Accuracy section below.

Here’s the current ESPHome YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173


esphome:
 name: emporia-vue2

external_components:
 - source: github://emporia-vue-local/esphome@dev
 components: [ emporia_vue ]

esp32:
 board: esp32dev
 framework:
 type: esp-idf
 version: recommended

# Enable Home Assistant API
mqtt:
 broker: mqtt.example.domain
 discovery_unique_id_generator: mac
 discovery_object_id_generator: none

logger:
 logs:
 sensor: INFO

ota:
 password: !secret ota
 num_attempts: 3

wifi:
 ssid: !secret wifi_ssid
 password: !secret wifi_password

i2c:
 sda: 21
 scl: 22
 scan: false
 frequency: 200kHz  # recommended range is 50-200kHz
 id: i2c_a

time:
 - platform: sntp
 id: my_time
 timezone: America/Los_Angeles

debug:
 update_interval: 120s

text_sensor:
 - platform: debug
 reset_reason:
 name: "Reset Reason"

# these are called references in YAML. They allow you to reuse
# this configuration in each sensor, while only defining it once
.defaultfilters:
 - &moving_avg
 # we capture a new sample every 0.24 seconds, so the time can
 # be calculated from the number of samples as n * 0.24.
 sliding_window_moving_average:
 # we average over the past 2.88 seconds
 window_size: 12
 # we push a new value every 1.44 seconds
 send_every: 6
 - &invert
 # invert and filter out any values below 0.
 lambda: 'return max(-x, 0.0f);'
 - &pos
 # filter out any values below 0.
 lambda: 'return max(x, 0.0f);'
 - &abs
 # take the absolute value of the value
 lambda: 'return abs(x);'
 # Reduce noise in the power class
 - &power_max
 or:
 - delta: 5
 - throttle: 60s
 - &power_min
 throttle: 3s
 - &throttle_energy
 or:
 - delta: 10
 - throttle: 60s

sensor:
 - platform: emporia_vue
 i2c_id: i2c_a
 phases:
 - id: phase_a  # Verify that this specific phase/leg is connected to correct input wire color on device listed below
 input: BLACK  # Vue device wire color
 calibration: 0.022 # 0.022 is used as the default as starting point but may need adjusted to ensure accuracy
 # To calculate new calibration value use the formula * / 
 voltage:
 name: "Phase A Voltage"
 filters: [*moving_avg, *pos]
 - id: phase_b  # Verify that this specific phase/leg is connected to correct input wire color on device listed below
 input: RED  # Vue device wire color
 calibration: 0.022 # 0.022 is used as the default as starting point but may need adjusted to ensure accuracy
 # To calculate new calibration value use the formula * / 
 voltage:
 name: "Phase B Voltage"
 filters: [*moving_avg, *pos]
 ct_clamps:
 - phase_id: phase_a
 input: "A" # Verify the CT going to this device input also matches the phase/leg
 power:
 name: "Phase A Power"
 id: phase_a_power
 device_class: power
 filters: [*moving_avg, *pos]
 - phase_id: phase_b
 input: "B" # Verify the CT going to this device input also matches the phase/leg
 power:
 name: "Phase B Power"
 id: phase_b_power
 device_class: power
 filters: [*moving_avg, *pos]
 # Pay close attention to set the phase_id for each breaker by matching it to the phase/leg it connects to in the panel
 # Some circuits are commented out because they don't have a CT connected yet
 - { phase_id: phase_a, input: "1", power: { name: "Heat Pump Power #1", internal: true, id: cir1, filters: [ *pos, multiply: 2 ] } }
 - { phase_id: phase_a, input: "2", power: { name: "Oven Power #2", id: cir2, internal: true, filters: [ *pos, multiply: 2 ] } }
 - { phase_id: phase_a, input: "5", power: { name: "Dryer Power #5", internal: true, id: cir5, filters: [ *pos, multiply: 2 ] } }
 - { phase_id: phase_a, input: "6", power: { name: "Dishwasher / Disposal Power #6", internal: true, id: cir6, filters: [ *pos ] } }
 - { phase_id: phase_b, input: "8", power: { name: "Kitchen Power #8", id: cir8, internal: true, filters: [ *pos ] } }
 - { phase_id: phase_a, input: "9", power: { name: "Washer Power #9", id: cir9, internal: true, filters: [ *pos ] } }
 - { phase_id: phase_a, input: "10", power: { name: "Kitchen Power #10", id: cir10, internal: true, filters: [ *pos ] } }
 # - { phase_id: phase_b, input: "11", power: { name: "Bathroom Power #11", id: cir11, filters: [ *moving_avg, *pos ] } }
 # - { phase_id: phase_b, input: "12", power: { name: "Stove / Hood Fan Power #12", id: cir12, filters: [ *moving_avg, *pos ] } }
 # - { phase_id: phase_a, input: "13", power: { name: "Microwave Power #13", id: cir13, filters: [ *moving_avg, *pos ] } }
 - { phase_id: phase_a, input: "14", power: { name: "Bedroom Power #14", id: cir14, internal: true, filters: [ *pos ] } }
 - { phase_id: phase_b, input: "15", power: { name: "General Power #15", internal: true, id: cir15, filters: [ *pos ] } }
 - { phase_id: phase_b, input: "16", power: { name: "General Power #16", internal: true, id: cir16, filters: [ *pos ] } }

 - { platform: copy, id: cir1_b, source_id: cir1, filters: [ *power_min, *power_max ], name: "Heat Pump Power #1" }
 - { platform: copy, id: cir2_b, source_id: cir2, filters: [ *power_min, *power_max ], name: "Ovean Power #2" }
 - { platform: copy, id: cir5_b, source_id: cir5, filters: [ *power_min, *power_max ], name: "Dryer Power #5" }
 - { platform: copy, id: cir6_b, source_id: cir6, filters: [ *power_min, *power_max ], name: "Dishwasher / Disposal Power #6" }
 - { platform: copy, id: cir8_b, source_id: cir8, filters: [ *power_min, *power_max ], name: "Kitchen Power #8" }
 - { platform: copy, id: cir9_b, source_id: cir9, filters: [ *power_min, *power_max ], name: "Washer Power #9" }
 - { platform: copy, id: cir10_b, source_id: cir10, filters: [ *power_min, *power_max ], name: "Kitchen Power #10" }
 - { platform: copy, id: cir14_b, source_id: cir14, filters: [ *power_min, *power_max ], name: "Bedroom Power #14" }
 - { platform: copy, id: cir15_b, source_id: cir15, filters: [ *power_min, *power_max ], name: "General Power #15" }
 - { platform: copy, id: cir16_b, source_id: cir16, filters: [ *power_min, *power_max ], name: "General Power #16" }


 - platform: template
 name: "Total Power"
 lambda: return id(phase_a_power).state + id(phase_b_power).state;
 update_interval: 1s
 id: total_power
 device_class: power
 state_class: measurement
 unit_of_measurement: "W"

 - name: "Total Daily Energy"
 power_id: total_power
 platform: total_daily_energy
 accuracy_decimals: 0
 restore: false
 unit_of_measurement: Wh
 state_class: total_increasing
 device_class: energy
 filters: [ *throttle_energy ]

 - { power_id: cir1, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Heat Pump Energy" }
 - { power_id: cir2, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Oven Energy" }
 - { power_id: cir5, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Dryer Energy" }
 - { power_id: cir6, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Dishwasher / Disposal Energy" }
 - { power_id: cir8, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Kitchen #8 Energy" }
 - { power_id: cir9, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Washer Energy" }
 - { power_id: cir10, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Kitchen #10 Energy" }
 - { power_id: cir14, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Bedroom Energy" }
 - { power_id: cir15, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "General #15 Energy" }
 - { power_id: cir16, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "General #16 Energy" }

Save the above configuration, then hit Install > Manual Install > Modern Format. Let it compile, then download. Then using the

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


python3 -m esptool --chip esp32 -p /dev/cu.usbserial-130 write_flash 0x0 ~/Downloads/emporia-vue2-factory.bin
esptool.py v4.4
Serial port /dev/cu.usbserial-130
Connecting....
Chip is ESP32-D0WD (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: a8:48:fa:97:90:3c
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00000000 to 0x000dbfff...
Compressed 897488 bytes to 564586...
Wrote 897488 bytes (564586 compressed) at 0x00000000 in 54.7 seconds (effective 131.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

References

https://flaviutamas.com/2021/reversing-emporia-vue-2

Installing CTs

Installing the CTs is the hardest and most dangerous part of this. If you’re not familiar with the risks with working inside a breaker box and the fact that the mains generally can’t be turned off, you should consult a qualified electrician.

The official hardware guide can be found here.

When installing the CTs, make sure that you match the phases with the colors and the holes in the top of the Vue2. Otherwise you won’t get the correct data. Note the bolded items. My BLACK wire went into the A hole and RED went into the B hole on the top of the Vue2.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


 - id: phase_a 
 input: **BLACK** # Vue device wire color
 - id: phase_b  # Verify that this specific phase/leg is connected to correct input wire color on device listed below
 input: **RED** # Vue device wire color
 ct_clamps:
 - phase_id: phase_a
 input: "**A**" # Verify the CT going to this device input also matches the phase/leg
 power:
 name: "Phase A Power"
 id: phase_a_power
 device_class: power
 filters: [*moving_avg, *pos]
 - phase_id: phase_b
 input: "**B**" # Verify the CT going to this device input also matches the phase/leg
 power:
 name: "Phase B Power"
 id: phase_b_power
 device_class: power
 filters: [*moving_avg, *pos]
 # Pay close attention to set the phase_id for each breaker by matching it to the phase/leg it connects to in the panel
 - { phase_id: **phase_a**, input: "1", power: { name: "Heat Pump Power #1", id: cir1, filters: [ *moving_avg, *pos, multiply: 2 ] } }

An in progress pic showing it installed with just the CTs on the main loads. Note that the third phase is unused and is joined to the neutral and not left floating.

When installing the CTs on the individual circuits in North American houses, you may encounter circuits that have two phases (240v circuits). Two CTs are needed if it’s unbalanced, but one CT is sufficient if it’s a “balanced load.” I found the following quote to help me when installing:

If a double breaker circuit is “balanced”, power is evenly drawn through both poles. In this instance, an energy monitoring app can typically take the reading from one CT and multiply it by 2 to get the correct power reading.

However, if a circuit is “unbalanced”, two CTs should be used. Pumps, electric resistance heat, and HVAC units are typically balanced. Subpanels, dryers, electric ovens/ranges, and hot tubs are not balanced. Typically, if a piece of equipment is doing more than one thing, it is an unbalanced load. A dryer needs to rotate the drum and dry the clothes. Also, if a circuit has a neutral wire, this most likely means that the load is unbalanced and requires two CTs.

https://www.powerwisesystems.com/blog/measure-electricity-use-current-transformers/

I only found one unbalanced load, my clothes dryer. I installed a CT on both legs, then updated the ESPHome template to aggregate both legs, then send it to MQTT. The following snippet shows how to implement an unbalanced load sensor:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


 ct_clamps:
 - { phase_id: phase_a, input: "5", power: { name: "Dryer Power #5", internal: true, id: cir5, filters: [ *pos ] } }
 - { phase_id: phase_b, input: "7", power: { name: "Dryer Power #7", internal: true, id: cir7, filters: [ *pos ] } }

[...]

 - platform: template
 name: "Dryer Power"
 lambda: return id(cir5).state + id(cir7).state;
 update_interval: 1s
 id: dryer_power
 device_class: power
 state_class: measurement
 unit_of_measurement: "W"
 filters: [ *power_min, *power_max ]

 - { power_id: dryer_power, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Dryer Energy" }

Next phase after installing almost all of the CTs. Not all circuits will get CTs.

After everything’s safely installed, turn the breaker on and verify that you’re receiving traffic in MQTT and in Home Assistant.

HA Recorder Management

When I first installed the Vue2 and pointed it to Home Assistant, it didn’t take long for the Postgres instance behind my HA Recorder to run out of disk space.

I checked the highest metrics and as I suspected, the integral was using the most space. It was being emitted every second as a watt-hour with 2 decimal points, but everywhere else I was using kilowatt-hours with 2 decimal points.

1
2
3
4
5
6
7


SELECT
 COUNT(*) AS cnt,
 COUNT(*) * 100 / (SELECT COUNT(*) FROM states) AS cnt_pct,
 entity_id
FROM states
GROUP BY entity_id
ORDER BY cnt DESC

# of states	% total	entity_id
643,609	22%	sensor.total_energy_3 (The integral generated by Vue2)
251,745	8%	sensor.total_power
251,281	8%	sensor.total_energy
251,129	8%	sensor.daily_electricity_usage
239,813	8%	sensor.total_energy_cost
210,866	7%	sensor.phase_b_power
192,576	6%	sensor.phase_a_power
92,212	3%	sensor.media_cabinet_total_power

Not good. This is likely due to the fact that the integration sensor in ESPHome emits a new value every 2.88s just like the Watt sensor. This metric is designed to be long-term and Home Assistant’s energy tab aggregates it up per hour, so there’s no need for that level of precision. Instead, I’m going to have it emit it less frequently, but first let’s see what else is wrong.

Accuracy Analysis and Comparison

I then tried to compare the accuracy of the Vue2 versus other devices I have been using prior to this. The following shows a comparison with the Zooz ZEN15 Z-Wave Outlet installed on my washing machine.

Baseline

Here we see there is a difference of about ~1W, but this was attributed to the power draw of the outlet itself combined with another low power device on the same circuit not measured. After removing those devices, the measured value converges on about 750mW from both devices. This is good and indicates both devices are at least agreeing with each other.

Under Load

However, under load from a single wash load we see quite a different perspective:

This graph shows a big difference in the wave-forms. The Z-Wave outlet reported 126Wh for the entire run vs the Emporia Vue2 reporting 204Wh for a difference of 80Wh. Zooming in on this wave form in two places:

I suspect this is caused by the power metric being averaged over ~3seconds and all the momentary drops and increases were getting averaged out.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


 - &moving_avg
 # we capture a new sample every 0.24 seconds, so the time can
 # be calculated from the number of samples as n * 0.24.
 sliding_window_moving_average:
 # we average over the past 2.88 seconds
 window_size: 12
 # we push a new value every 1.44 seconds
 send_every: 6

# [...]

 - { phase_id: phase_a, input: "9", power: { name: "Washer Power #9", id: cir9, filters: [ *moving_avg, *pos ] } }

I confirmed this by removing the moving_avg filter and configured my Z-Wave outlet to send updates every 1W change and did another load of laundry.

To verify, I ran a query to total the watt-hours over the entire run-time:

1
2
3
4
5
6


from(bucket: "homeassistant")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["_measurement"] == "W")
 |> filter(fn: (r) => r["entity_id"] == "washer_outlet_power" or r["entity_id"] == "washer_power")
 |> filter(fn: (r) => r["_field"] == "value")
 |> integral(unit: 1h)

Gave me 75Wh from the Vue2 vs 77Wh from the Z-Wave, a difference of <3% and the wave forms look like this:

These wave forms look a lot more similar than before.

Wave Form with large difference. Possibly caused by sampling or due to apparent/real power

To fix this, I changed how all sensors worked. Previously, esphome would perform a 2.88s moving average, then integrate that value for energy consumption. However, in spiky situations, that could become quite inaccurate. Trying to use HA’s helpers to integrate this would require the esp to send a huge power updates very frequently and increase Recorder usage.

Diagram showing the ESPHome component data flow before and after.

Instead, I could tell ESPHome to integrate the raw value without any averaging and throttle both the power and energy sensor updates to something reasonable. I marked the raw sensors as internal: true, then used the copy component to send the power updates every 3s - 60s. Three seconds ensures we don’t send updates too quickly, and between 3s and 60s if the power fluctuates by >10W, then it’ll send an update, and if not it sends an updates max every 60s. This configuration provides a nice trade-off so constant, lower power devices don’t send a lot of updates.

The energy component takes the raw value and performs the integration, then sends it every 60s since there’s very little value in sending more frequently.

The following snippet shows how a single circuit is represented in the template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


.defaultfilters:
 # Reduce noise in the power class
 - &power_max
 or:
 - delta: 5
 - throttle: 60s
 - &power_min
 throttle: 3s
 - &throttle_energy
 throttle: 60s

# [...]

sensor:
 - platform: emporia_vue
 [...]
 ct_clamps:
 [...]
 - { phase_id: phase_b, input: "16", power: { name: "General Power #16", internal: true, id: cir16, filters: [ *pos ] } }

 - { platform: copy, id: cir16_b, source_id: cir16, filters: [ *power_min, *power_max ], name: "General Power #16" }

 - { power_id: cir16, platform: total_daily_energy, accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "General #16 Energy" }

Voltage Calibration

Each phase has a voltage calibration constant value that influences how the Vue2 measures the voltage. I don’t know how this influences the current or power sensors, but let’s get it calibrated just to be sure.

Using a volt meter, I initially tried to measure the voltage of each phase inside the breaker box connecting the negative to the neutral bar and positive to each phase main line, however the measured voltage fluctuated depending on whether the panel was open or closed. Opened when I measured it, the volt meter agreed with what was in HA, but then I closed the panel and it’d deviate. Notice the first two blue lines in the figure above. I don’t know what could cause this.

After that, I instead measured the voltage from two outlets on each phase without the panel being open. I used the formula as mentioned in the template for each phase:

1
2


# 0.022 is used as the default as starting point but may need adjusted to ensure accuracy
# To calculate new calibration value use the formula <in-use calibration value> * <accurate voltage> / <reporting voltage>

{Initial Calibration Value} \times \frac{{Measured Volts}}{{Reported Volts}} = {New Calibration Value}

I plugged in my values and get this:

0.022 \times \frac{123.0 v}{126.5 v} = 0.022737

Then plugged both values into the template and redeployed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


sensor:
 - platform: emporia_vue
 i2c_id: i2c_a
 phases:
 - id: phase_a
 input: BLACK
 calibration: 0.022737
 voltage:
 name: "Phase A Voltage"
 filters: [*moving_avg, *pos]
 - id: phase_b
 input: RED
 calibration: 0.021295
 voltage:
 name: "Phase B Voltage"
 filters: [*moving_avg, *pos]

After that the measured voltages seemed to align with my volt meter and converged together (not that both phases need to match.)

Comparison with Electric Provider

Next up, it’s time to compare it with my utility provider. This should match as close as possible so ensure that the numbers I show in Home Assistant are actually reflective of reality. My provider shows a daily energy breakdown in my account page (though they round the numbers and I had to use the browser dev tools to see the raw numbers.)

Overall pretty close with a difference of -8.26kWh over the last 30 days and ~2.8%. Not perfect, but I’ll continue to monitor and look for opportunities to improve accuracy.

Conclusion

In this post, I walked through how to use the Emporia Vue2 to monitor my whole home’s energy usage, some different strategies to improve accuracy and reduce MQTT traffic.

Zeppelin v0.10 not showing matplotlib graphs

Sat, 18 Feb 2023 00:00:00 +0000

I upgraded to Apache Zeppelin v0.10.x from v0.9.x and randomly my Python Matplotlib scripts stopped rendering images. Anything that called the plot method would just return the string response of the function. Like below:

1
2
3
4
5


%python
import matplotlib.pyplot as plt
plt.plot([1, 2, 3])

[<matplotlib.lines.Line2D at 0x7ff547624210>]

If this happens to you, just add the following directive after %python:

1
2
3
4
5


%python
%matplotlib inline

import matplotlib.pyplot as plt
plt.plot([1, 2, 3])

After that, it should work again:

I’m not sure why this seems to happen. After it’s applied it seems to apply to all future blocks too.

Rewriting Home Assistant Long-term statistics from InfluxDB

Thu, 01 Dec 2022 00:00:00 +0000

In an earlier post, I made an error that incorrectly aggregated the energy data which resulted in hugely inflated aggregated energy usage. All the un-aggregated data was accurate, but the sums were wrong. Luckily I had all the raw data stored in InfluxDB and could rebuild it.

In this post, I walk through how to re-write the Home Assistant Long-term statistics database to fix this mistake.

A grossly high electric bill

First, I opened up the Influx UI and constructed a query to generate the corrected metrics. The HA database requires two fields: sum and state. One describes the cumulative running sum, and the other is the sum at the end of the hour. I didn’t fully understand how sum differed from state, so I just assumed they were the same.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


from(bucket: "energy-1h")
 |> range(start: -365d, stop: now())
 |> filter(fn: (r) => r["_measurement"] == "kWh")
 |> filter(fn: (r) => r["_field"] == "value")
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> filter(fn: (r) => r["entity_id"] != "main_panel_total_energy" and r["entity_id"] != "sub_panel_32_total_power" and r["entity_id"] != "daily_electricity_usage")
 |> group(columns: ["_measurement", "_field"])
 |> aggregateWindow(every: 1h, fn: sum, createEmpty: false)
 |> cumulativeSum()
 |> yield(name: "mean")

I ran this query, then clicked download to CSV.

That gave me a file that looked like the following:

1
2
3
4
5
6


#group,false,false,true,true,false,false,true,true,true,true
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string
#default,mean,,,,,,,,,
,result,table,_start,_stop,_time,_value,_field,_measurement,domain,entity_id
,,0,2022-08-01T06:08:15Z,2022-12-01T07:08:15.162Z,2022-08-21T09:00:00Z,2.569999999999993,value,kWh,sensor,main_panel_total_energy
,,0,2022-08-01T06:08:15Z,2022-12-01T07:08:15.162Z,2022-08-21T10:00:00Z,1.7900000000000063,value,kWh,sensor,main_panel_total_energy

Delete first 3 lines and one of the two trailing lines.

1

$ tail -n+4 '2022-11-30_23 38_influxdb_data.csv' | head -n-1 > influxdata.csv

Make sure the file is nearby on the host running HomeAssistant, then stop HA and open up the file:

1
2
3


# sqlite3 home-assistant_v2.db
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.

I looked at my target table schema:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


sqlite> .schema statistics
CREATE TABLE statistics (
 id INTEGER NOT NULL,
 created DATETIME,
 start DATETIME,
 mean FLOAT,
 min FLOAT,
 max FLOAT,
 last_reset DATETIME,
 state FLOAT,
 sum FLOAT,
 metadata_id INTEGER,
 PRIMARY KEY (id),
 FOREIGN KEY(metadata_id) REFERENCES statistics_meta (id) ON DELETE CASCADE
);
CREATE UNIQUE INDEX ix_statistics_statistic_id_start ON statistics (metadata_id, start);
CREATE INDEX ix_statistics_start ON statistics (start);
CREATE INDEX ix_statistics_metadata_id ON statistics (metadata_id);

Then import the CSV file into the database

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


sqlite> .mode csv
sqlite> .import "../influxdata.csv" influx
sqlite> .schema statistics
sqlite> .schema influx
CREATE TABLE influx(
 "" TEXT,
 "result" TEXT,
 "table" TEXT,
 "_field" TEXT,
 "_measurement" TEXT,
 "_start" TEXT,
 "_stop" TEXT,
 "_time" TEXT,
 "_value" TEXT,
 "domain" TEXT,
 "entity_id" TEXT,
 "state" TEXT
);

Now we’re reading to replace the data. First, find the id of the entity to replace. The first column will be the id

1
2
3
4


sqlite> select * from statistics_meta where statistic_id = 'sensor.main_panel_total_energy';
1,sensor.main_panel_total_energy,recorder,kWh,0,1,

sqlite> delete from statistics where metadata_id = 1;

Then insert the data and drop the temp table

1
2
3


sqlite> insert into statistics select null AS id, datetime(_time) as created, datetime(_time, '-1 hour') as start, null as mean, null as min, null as max, null as last_reset, cast(_value as float) as state, cast(_value as float) as sum, 1 as metadata_id from influx;

sqlite> drop table influx;

Then delete the history for the energy_cost because at the end of the next hour, HA will distort the metrics again:

1

sqlite> delete from states where entity_id = 'sensor.main_panel_total_power' or entity_id = 'sensor.main_panel_total_energy' or entity_id = 'sensor.main_panel_total_energy_cost';

Home Assistant stores some intermediary data in the statistics_short_term table. This may cause problems and I’ve had to delete those records sometimes

1

sqlite> delete from statistics_short_term where metadata_id = 1;

Then open home-assistant/.storage/core.restore_state and change the entry for main_panel_total_energy to match the latest value from InfluxDB.

Restart Home Assistant.

Voilà. The data is corrected:

From a random Kubernetes control plane crash to a new RAID array

Mon, 14 Nov 2022 00:00:00 +0000

My external cluster runs on 3 different dedicated servers (most from SoYouStart.com.) I have 3 machines since the Kubernetes control plane needs 3 or more to be able to have a quorum and be able to handle any one machine going down. If one machine goes down, then the other two maintain a majority and can agree on the state of the cluster.

I randomly encountered issues where the Kubernetes control plane of Rancher UI would crash and restart. While this cluster didn’t really matter, it still annoyed me and wanted to figure it out.

I narrowed it down to one single host and documented the steps I took to resolve this issue which seems to be have been caused by one machine using HDDs and all other hosts using SSDs.

The Symptoms

The first thing that I did was looked at all the hosts and noticed that one host (srv6) showed that a few Kubernetes control plane containers were restarting.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


ccb77f64413b rancher/hyperkube:v1.24.4-rancher1 "/opt/rke-tools/entr…" 5 weeks ago Up 24 hours kube-proxy
08ec5c6161eb rancher/hyperkube:v1.24.4-rancher1 "/opt/rke-tools/entr…" 5 weeks ago Up 24 hours kubelet
88eaa48e458f rancher/hyperkube:v1.24.4-rancher1 "/opt/rke-tools/entr…" 5 weeks ago Up 15 hours kube-scheduler
4d603be7b4ef rancher/hyperkube:v1.24.4-rancher1 "/opt/rke-tools/entr…" 5 weeks ago Up 4 hours kube-controller-manager
dc86384410c2 rancher/hyperkube:v1.24.4-rancher1 "/opt/rke-tools/entr…" 5 weeks ago Up 24 hours kube-apiserver
05df16372b51 rancher/mirrored-coreos-etcd:v3.5.4 "/usr/local/bin/etcd…" 5 weeks ago Up 24 hours etcd

$ docker inspect kube-controller-manager
[
 {
 ...
 "State": {
 "StartedAt": "2022-11-12T02:30:35.889344997Z",
 "FinishedAt": "2022-11-12T02:30:31.970955836Z"

I was already running Grafana Loki, which aggregated all of my Docker container logs from all machines and zoomed into the time when the container last restarted:

{docker_container=~"/kube-apiserver|/kube-controller-manager"} |= ``

I scrolled through the logs and started seeing some smoke. A request to the kube-apiserver was timing out. kube-controller-manager calls kube-apiserver, which then calls etcd.

1
2
3
4


2022-11-11T18:30:25-08:00 E1112 02:30:25.799867 1 leaderelection.go:330] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
...
2022-11-11T18:30:26-08:00 Trace[1355871407]: [5.002442529s] [5.002442529s] END
2022-11-11T18:30:26-08:00 E1112 02:30:26.701570 1 timeout.go:141] post-timeout activity - time-elapsed: 6.504883ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager" result: <nil>

Looking in the etcd logs, I found some log statements saying “apply request took too long”. According to the etcd docs, this error statement is printed when the average time to write a change to disk exceeds 100ms. Since at least 2 hosts need to acknowledge a write for quorum, this caused these writes to slow down.

I searched across the hosts for this log statement and found that one particular host (srv6) gave this error far more than any other host.

1

sum by(host) (count_over_time({docker_container="/etcd"} |= `apply request took too long` [1h]))

The biggest difference between these hosts is that everything else has SSDs, but this one has spinning rust HDDs (4x 2TB HDDs in RAID1.) Is the hard drive dying?

Hard Drive Problems?

Next, I investigated further to see if there’s something wrong with the drives. The following Prometheus query gets the average write latency of I/O to the disk.

1

sum by(instance) (increase(node_disk_write_time_seconds_total[1m])) / sum by(instance) (increase(node_disk_writes_completed_total[1m]))

Here we see this host showing spikes in write latency every 30 minutes to the md2 disk, but my sd[a-d] drives show reasonable latency: <50ms. The md2 is a RAID1 array that mirrors to sda, sdb, sdc, and sdd.

A couple ideas come to mind:

There’s a spike every 30 minutes with smaller spikes every 5 minutes. Possibly caused by scheduled jobs
Only md2 is spiking, but sd[a-d] are not spiking as much. Not sure why the RAID virtual array spikes if the underlying drives are not seeing a similar spike.

Okay by what’s md2?

md2 is a Linux software RAID setup. It’s configured at RAID 1 across all 4 drives. When I first setup this machine, I didn’t put much thought into what kind of RAID setup I wanted, I just picked some default values in the server UI and this is what I ended up with.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


mdadm --detail /dev/md2
/dev/md2:
 Version : 1.2
 Creation Time : Tue Dec 7 07:24:59 2021
 Raid Level : raid1
 Array Size : 1952332800 (1861.89 GiB 1999.19 GB)
 Used Dev Size : 1952332800 (1861.89 GiB 1999.19 GB)
 Raid Devices : 4
 Total Devices : 4
 Persistence : Superblock is persistent

 Intent Bitmap : Internal

 Update Time : Sun Nov 13 02:48:53 2022
 State : active
 Active Devices : 4
 Working Devices : 4
 Failed Devices : 0
 Spare Devices : 0

Consistency Policy : bitmap

 Name : md2
 UUID : d49251e3:7091f035:ef641ea4:bff15c7f
 Events : 78807

 Number Major Minor RaidDevice State
 0 8 34 0 active sync /dev/sdc2
 1 8 50 1 active sync /dev/sdd2
 2 8 2 2 active sync /dev/sda2
 3 8 18 3 active sync /dev/sdb2

Checking SMART

Alright so nothing stands out in the RAID array yet. We can check the health of the drives using SMART using smartmontools.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95


sudo apt-get install smartmontools

sudo smartctl -a /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-52-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
...

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
 was suspended by an interrupting command from host.
 Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
 without error or no self-test has ever
 been run.
Total time to complete Offline
data collection: ( 24) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
 Auto Offline data collection on/off support.
 Suspend Offline collection upon new
 command.
 Offline surface scan supported.
 Self-test supported.
 No Conveyance Self-test supported.
 Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
 power-saving mode.
 Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
 General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 309) minutes.
SCT capabilities: (0x003d) SCT Status supported.
 SCT Error Recovery Control supported.
 SCT Feature Control supported.
 SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
 1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 1
 2 Throughput_Performance 0x0005 137 137 054 Pre-fail Offline - 79
 3 Spin_Up_Time 0x0007 216 216 024 Pre-fail Always - 314 (Average 264)
 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 105
 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
 8 Seek_Time_Performance 0x0005 142 142 020 Pre-fail Offline - 25
 9 Power_On_Hours 0x0012 092 092 000 Old_age Always - 59245
 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 104
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 225
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 225
194 Temperature_Celsius 0x0002 176 176 000 Old_age Always - 34 (Min/Max 20/52)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 51074 -
# 2 Short offline Completed without error 00% 51066 -
# 3 Short offline Completed without error 00% 51065 -
# 4 Short offline Completed without error 00% 51065 -
# 5 Short offline Completed without error 00% 51052 -
# 6 Short offline Completed without error 00% 51044 -
# 7 Short offline Completed without error 00% 51044 -
# 8 Short offline Completed without error 00% 51036 -
# 9 Short offline Completed without error 00% 51035 -
#10 Short offline Completed without error 00% 50512 -
#11 Short offline Completed without error 00% 50495 -
#12 Short offline Completed without error 00% 50487 -

SMART Selective self-test log data structure revision number 1
 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
 1 0 0 Not_testing
 2 0 0 Not_testing
 3 0 0 Not_testing
 4 0 0 Not_testing
 5 0 0 Not_testing
Selective self-test flags (0x0):
 After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Checking all four hosts didn't show any SMART concerns and said, while they're older, they're still healthy.

Scheduled Jobs?

What about the regular spikes on the clocks of every 30 minutes and smaller ones every 5 minutes. My host will be running scheduled jobs in Kubernetes and possibly in crontab. Let’s take a look at Kubernetes first:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


kubectl --context=local get cronjobs --all-namespaces
NAMESPACE NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
longhorn-system daily-backup 43 10 * * * False 0 16h 70d
longhorn-system weekly-backup 0 2 * * SUN False 0 53m 70d
test testjob */30 * * * * False 0 23m 237d
modtalk wordpress-cron */5 * * * * False 0 171m 119d
openzipkin zipkin-dependencies 25 5/6 * * * False 0 3h28m 82d
testjob regular-job 0 3/12 * * * False 0 11h 237d
sentry cleanup */5 * * * * False 0 168m 237d
sentry snuba-cleanup */5 * * * * False 0 8h 237d
sentry snuba-transactions-cleanup */5 * * * * False 0 8h 237d
technowizardry wordpress-cron */5 * * * * False 0 51m 237d

There were a number of jobs running every 5 minutes. I changed them to run less frequently and saw a reduction in write latency during the 5 minute spikes, but still saw spikes every time a CronJob ran.

Great so, the problem is because a CronJob starting is causing a large amount of disk I/O and slowing down writes for etcd.

Proposal

My hypothesis based on the above information is that the hard drives are getting overloaded with the amount of write or read write work meaning that higher priority I/O from Kubernetes’ etcd system is taking too long and is causing problems.

RAID1 will mirror the hard drive contents across all four hard drives. This is good for read performance since it can balance reads across all 4 drives, but writes slow down to the slowest drive. In a 4 drive system, this can be problematic.

Instead, I’m going to break the 4 drive mirror array into two, two-drive RAID1 arrays and move certain performance critical data to the second array and leave the first one for

Splitting RAID

Following a guide, I took out two drives from the existing array.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88


mdadm --manage /dev/md2 --fail /dev/sdc2
mdadm: set /dev/sdc2 faulty in /dev/md2
mdadm --manage /dev/md2 --fail /dev/sdd2
mdadm: set /dev/sdd2 faulty in /dev/md2

mdadm --manage /dev/md2 --remove /dev/sdc2
mdadm: hot removed /dev/sdc2 from /dev/md2

mdadm --manage /dev/md2 --remove /dev/sdd2
mdadm: hot removed /dev/sdd2 from /dev/md2

mdadm --grow -n 2 /dev/md2
raid_disks for /dev/md2 set to 3
mdadm --detail /dev/md2
/dev/md2:
 Version : 1.2
 Creation Time : Tue Dec 7 07:24:59 2021
 Raid Level : raid1
 Array Size : 3
 Used Dev Size : 1952332800 (1861.89 GiB 1999.19 GB)
 Raid Devices : 2
 Total Devices : 2
 Persistence : Superblock is persistent

 Intent Bitmap : Internal

 Update Time : Sun Nov 13 05:07:35 2022
 State : clean
 Active Devices : 2
 Working Devices : 2
 Failed Devices : 0
 Spare Devices : 0

Consistency Policy : bitmap

 Name : md2
 UUID : d49251e3:7091f035:ef641ea4:bff15c7f
 Events : 79300

 Number Major Minor RaidDevice State
 2 8 2 0 active sync /dev/sda2
 3 8 18 1 active sync /dev/sdb2

## Creating the new array

Next, I zeroed out the drives and created a new array called md3.

mdadm --zero-superblock /dev/sdc2
mdadm --zero-superblock /dev/sdd2

mdadm --create --verbose /dev/md3 --level=1 --raid-devices=2 /dev/sdc2 /dev/sdd2
mdadm: Note: this array has metadata at the start and
 may not be suitable as a boot device. If you plan to
 store '/boot' on this device please ensure that
 your boot-loader understands md/v1.x metadata, or use
 --metadata=0.90
mdadm: size set to 1952331776K
mdadm: automatically enabling write-intent bitmap on large array
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md3 started.

$ vi /etc/mdadm.conf
ARRAY /dev/md/md2 metadata=1.2 UUID=d49251e3:7091f035:ef641ea4:bff15c7f name=md2
ARRAY /dev/md/md3 metadata=1.2 UUID=10f87b98:e31749b4:ee14c8f0:5fae7870 name=md3

$ update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.15.0-52-generic
I: The initramfs will attempt to resume from /dev/sdd3
I: (UUID=e7159a7b-f2b7-4253-bfcd-80d78bb7df9c)
I: Set the RESUME variable to override this.

Then waited for it to rebuild the RAID array which took all night to complete.

mdadm --detail /dev/md3

Consistency Policy : bitmap

 Resync Status : 0% complete

 Name : md3 (local to host srv6.technowizardry.net)
 UUID : 10f87b98:e31749b4:ee14c8f0:5fae7870
 Events : 11

Sometime through this rebuild, the primary disk got mounted as read-only and couldn't be remounted as read-write which caused problems like, but a reboot fixed this

which service
bash: /usr/bin/which: Input/output error

I let it sync overnight and finally it was fully completed. Next we need to create an ext4

mkfs.ext4 /dev/md3

Folder Planning

I then identified which folders I wanted to move over to the new array. I started with just the bare minimum to see what effect it would have on performance:

/etc/kubernetes
/var/lib/etcd

Creating the new mount

Next, I identified the UUID of the new RAID array so I can enable auto mounting.

1
2
3


blkid
...
/dev/md3: UUID="af2d8991-0995-4aef-b5c8-78ca736eb664" BLOCK_SIZE="4096" TYPE="ext4"

Next, stop the running containers, move the folders over

1
2
3
4
5
6
7
8
9


cd /mnt/second
mkdir -p ./etc/kubernetes
mkdir -p ./var/lib/etcd

mv /etc/kubernetes /mnt/second/etc/
mv /var/lib/etcd /mnt/second/var/lib/etcd

mount -o bind /mnt/second/etc/kubernetes/ /etc/kubernetes/
mount -o bind /mnt/second/var/lib/etcd /var/lib/etcd

I changed the /etc/fstab file to ensure that Linux redirects the folders to the new array correctly:

1
2
3
4
5


$ vi /etc/fstab

UUID=af2d8991-0995-4aef-b5c8-78ca736eb664 /mnt/second ext4 defaults 0 2
/mnt/second/var/lib/etcd /var/lib/etcd none bind
/mnt/second/etc/kubernetes /etc/kubernetes none bind

Finally, restart all containers and watch as the Kubernetes node comes back online. After some time, Kubernetes stabilized and I saw a large reduction in the number of slow writes that etcd complained about. Unfortunately it didn’t hit zero, but sadly HDDs are slow compared to SSDs.

Loki search results for “apply request took too long”

Conclusion

In this post, I walked through identifying a Kubernetes control plane issue to figuring out it’s likely caused by disk write latency on one of the machines. This lead me to redesign the RAID array configuration. It improved things, but not as much as I hoped.

Visualizing Home Energy Usage in InfluxDB and Home Assistant

Sat, 15 Oct 2022 00:00:00 +0000

In previous posts in this series, I walked through how to get data flowing into Home Assistant.

In this post, we’ll get it flowing into InfluxDB for long-term retention.

InfluxDB

If you haven’t already installed InfluxDB, follow the official guide here. Then install the HA InfluxDB integration. The default InfluxDB integration sends a large amount of information that’s not useful. Here’s an example configuration that reduces it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


influxdb:
 host: influxdb-influxdb2.datastore.svc.cluster.local.
 port: 8086
 ssl: false
 api_version: 2
 token: {token}
 bucket: homeassistant
 organization: influxdata
 component_config_domain:
 sensor:
 ignore_attributes:
 - attribution
 - device_class
 - state_class
 - last_reset
 - integration
 - description
 - unit_of_measurement
 - friendly_name
 - type
 include:
 domains:
 - sensor

Queries

I’ve put together a few example queries that I use in my Grafana dashboards here.

Energy Usage using KwH

The following query returns energy usage over time. This query is more useful than simply averaging Watts over a time since it accounts for spikes and drops smaller than the window period.

Note: The InfluxDB seems to use the UTC time zone, whereas Home Assistant will use your local time zone on the energy dashboard. This can result in the numbers not matching, but using timeShift will fix them. Update timeZoneOffset to match your time zone.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


import "timezone"

option location = timezone.location(name: "America/Los_Angeles")

from(bucket: "homeassistant")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["_measurement"] == "kWh")
 |> filter(fn: (r) => r["_field"] == "value")
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> difference(nonNegative: true)
 |> aggregateWindow(every: v.windowPeriod, fn: sum)
 |> yield(name: "energy")

Energy Usage using watts

The prior query is more accurate if your sensors themselves report energy usage because the devices do continual aggregation, but the GreenEye Monitor does not itself collect kWh so we have to do the integral ourselves.

The following query returns energy usage over time. This query is more useful than simply averaging Watts over a time since it accounts for spikes and drops smaller than the window period.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


from(bucket: "homeassistant")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["_measurement"] == "W")
 |> filter(fn: (r) => r["_field"] == "value")
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> aggregateWindow(
 every: v.windowPeriod,
 fn: (tables=<-, column) =>
 tables
 |> integral(unit: v.windowPeriod)
 |> map(fn: (r) => ({ r with _value: r._value / 1000.0}))
 |> set(key: "_measurement", value: "kWh")
 )
 |> yield(name: "mean")

Energy Costs

This query calculates the cost of a given energy consumer over a time period and can be aggregated to any time period. If you zoom in far, you’ll find that things are measured in fractions of pennies.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


costPerkWh = 0.1056

from(bucket: "homeassistant")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> filter(fn: (r) => r["_measurement"] == "W")
 |> filter(fn: (r) => r["_field"] == "value")
 |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: true)
 |> fill(usePrevious: true)
 |> map(fn: (r) => ({r with
 _value: r._value * costPerkWh
 }))
 |> yield(name: "mean")

Data Retention

The kWh metric is important to know how much energy you’re actively using and being billed for, however Home Assistant tracks it as an ever increasing total number from 0 to infinity. This is problematic because you can’t aggregate a counter over different time periods to see how much you’re using month over month or week over week.

Instead, we need to convert this to a gauge-style metric where it contains the number of watt-hours used in a particular time period, not ever increasing.

I initially tried to aggregate to different levels including daily, however I encountered time zone issues (reference). InfluxDB runs all jobs in UTC, but I want to work in the local zone because that’s how my bill is calculated. Instead storing at hourly already provides a massive reduction in data sizes ~10k-15k data points per entity per day to only 24 per entity.

Data retention is controlled at the bucket level, so create a new bucket with a longer retention policy and name it something like energy-1hr (since it’ll store hourly aggregations.)

Seeing Double

I created a downsample job based on the kWh query described ahead, but after running the job for a few days, I tried comparing it to the raw data stored in my original bucket and found that everything seemed to be shifted over by 1 hour.

But after shifting the query over, I still saw differences. I wrote a query to compare the results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


import "timezone"
import "date"
import "join"
import "math"

option location = timezone.location(name: "America/Los_Angeles")

DOWNSAMPLED = from(bucket: "energy-1h")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["_measurement"] == "kWh")
 |> filter(fn: (r) => r["_field"] == "value")
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> filter(fn: (r) => r["entity_id"] == "total_energy")
 |> aggregateWindow(every: 1h, fn: sum, createEmpty: false)
 |> drop(columns: ["_start", "_stop"])
 |> group(columns: ["entity_id"])

RAW = from(bucket: "homeassistant")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["_measurement"] == "kWh")
 |> filter(fn: (r) => r["_field"] == "value" and r["entity_id"] == "total_energy")
 |> difference(nonNegative: true)
 |> aggregateWindow(every: 1h, fn: sum, createEmpty: false)
 |> drop(columns: ["_start", "_stop"])
 |> group(columns: ["entity_id"])

COMPUTED = RAW
 |> timeShift(duration: -1s)
 |> aggregateWindow(every: 1h, fn: sum, createEmpty: false)
 |> drop(columns: ["_start", "_stop"])
 |> group(columns: ["entity_id"])

join.left(left: RAW, right: DOWNSAMPLED, on: (l, r) => l._time == r._time, as: (l, r) => ({ l with _value: r._value - l._value, _field: "error" }))
 |> set(key: "_field", value: "error")
 |> map(fn: (r) => ({r with _value: math.abs(x: r._value)}))
 |> sum(column: "_value")

Which gave me a total error of 4.93 kWh / 7 days. This is weird. I started researching bugs on the internet and came across these:

https://github.com/influxdata/influxdb/issues/17100 - Task alignment issue

https://github.com/influxdata/influxdb/issues/17323 - Task Alignment issue

https://github.com/influxdata/flux/issues/1730 - aggregateWindow _start/_stop issue

After some investigation, I found out that aggregateWindow() rounds windows to the end of the window. This meant that data from 10am-11am was placed into a record with _time:11am, then the second aggregateWindow when I pulled it out of the downsampled bucket, windowed it to be 12pm. I couldn’t just drop the second aggregateWindow function because I wanted to be able to aggregate up into larger buckets if needed.

My solution for this is to timeShift the data by -1s:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


BY_KWH =
 from(bucket: "homeassistant")
 |> range(start: -task.every)
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> filter(fn: (r) => r["_measurement"] == "kWh")
 |> filter(fn: (r) => r["_field"] == "value")
 |> difference(nonNegative: true)

BY_KWH
 |> aggregateWindow(every: 1h, fn: sum, createEmpty: false)
 |> timeShift(duration: -1s)
 |> to(bucket: "energy-1h")

This brought the error down to 0.25 kWh across the last 7 days.

Unexpected differences

Even after temporarily adjusting the data and comparing, I still noticed errors. . My thought was that possible the first item was not being calculated correctly because the difference() was ignoring the first item or not correctly missing the first few minutes of energy usage. Switching to show the time in UTC seemed to confirm this issue:

I fixed this by aggregating the last 2 hours of data up to calculate the difference over the last midnight, then saving only the last 1 hour:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


BY_KWH =
 from(bucket: "homeassistant")
 |> range(start: -2h)
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> filter(fn: (r) => r["_measurement"] == "kWh")
 |> filter(fn: (r) => r["_field"] == "value")
 |> difference(nonNegative: true)

BY_KWH
 |> aggregateWindow(every: 1h, fn: sum, createEmpty: false)
 |> range(start: -1h)
 |> timeShift(duration: -1s)
 |> to(bucket: "energy-1h")

That helped, but looking at the raw kWh graph, there didn’t seem to be a lot of precision in the data. To fix this, I increased the precision of the Home Assistant Helper.

Four decimal precision points ensures that we can measure loads as low as 100mWh. Probably over kill, but I’ve got some devices that idle there.

This increased the number of events flowing into InfluxDB and the Home Assistant Recorder, but the aggressive downsampling will compensate:

Create a InfluxDB Job

That leads to the final query that I scheduled in the Influx UI under Tasks > Create Task:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54


import "timezone"
import "strings"
import "date"

option location = timezone.location(name: "America/Los_Angeles")
option task = {name: "Energy Downsample (Hourly)", every: 1h, offset: 2m}

start = date.add(d: -3h, to: now())
actual = -2h

BY_KWH =
 from(bucket: "homeassistant")
 |> range(start: start)
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> filter(fn: (r) => r["_measurement"] == "kWh")
 |> filter(fn: (r) => r["_field"] == "value")
 |> difference(nonNegative: true, keepFirst: true)

BY_KWH
 |> aggregateWindow(every: 1h, fn: sum, createEmpty: false)
 |> range(start: actual)
 |> drop(columns: ["_start", "_stop"])
 |> to(bucket: "energy-1h")

// Roll-up Brultech Energy Data
BY_WATTSEC =
 from(bucket: "homeassistant")
 |> range(start: start)
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> filter(fn: (r) => r["_field"] == "watt_seconds")
 |> difference(nonNegative: true)

BY_WATTSEC
 |> aggregateWindow(every: 1h, fn: sum, createEmpty: false)
 |> range(start: actual)
 |> map(fn: (r) => ({r with _value: r._value / 1000.0 / 60.0 / 60.0}))
 |> set(key: "_field", value: "value")
 |> set(key: "_measurement", value: "kWh")
 |> to(bucket: "energy-1h")

COST =
 from(bucket: "homeassistant")
 |> range(start: start)
 |> filter(fn: (r) => r["_measurement"] == "USD")
 |> filter(fn: (r) => r["_field"] == "value")
 |> filter(fn: (r) => r["domain"] == "sensor")
 |> filter(fn: (r) => strings.hasSuffix(v: r["entity_id"], suffix: "_energy_cost"))
 |> difference(nonNegative: true)

COST
 |> aggregateWindow(every: 1h, fn: sum, createEmpty: false)
 |> range(start: actual)
 |> drop(columns: ["_start", "_stop"])
 |> to(bucket: "energy-1h")

Over-engineering a home air quality dashboard

Mon, 29 Aug 2022 00:00:00 +0000

Air—it’s invisible, I can’t see it, but I feel effects of it in so many ways, temperature, humidity, gas composition, but I lacked sensors to measure it. In this post, I walk through some different Air Quality sensors that I found and how I wired them up into a dashboard.

Prerequisite Software

This project depends on a few software components. This post will assume that you have these set up already.

HomeAssistant - Home Automation central system manages sensor lifecycle
MQTT Broker - Message broker for ingesting sensor data
InfluxDB - Time series database for short and medium-term data retention
Grafana (Optional) - Used for building complex visualization dashboards
Node Red - Graphical programming environment that I use to process sensor data
Node Red Home Assistant Addon - Needs to be installed to create HA entities
ESPHome - Required for DIY built sensors

Architecture Diagram

My system architecture for collecting and storing environmental sensor data looks like this:

A Quick Intro to AQI Metrics

AQI stands for Air Quality Index. It doesn’t actually represent a singular consistent metric and different organizations and governments calculate their indexes slightly different. Popular ones include the EPA AQI, AQandU

More information on the different types of indexes can be found here.

Air quality sensors don’t directly calculate the AQI and instead track low level metrics such as PM1.0, PM2.5, PM10, etc. PM2.5 corresponds to the amount of Particulate Matter that is 2.5μm or smaller.

Courtesy of the CA Air Resources Board

PM10 includes dust, pollen, and mold, whereas PM2.5 starts to identify soot and smoke. Note that there are many particles that are smaller that can’t be easily measured with most home sensors. Some sensors (like the custom sensor I have) may end up estimating PM1.0 counts from the PM2.5 counts.

Given this, we’ll need the formulas that each one uses to be able to calculate the AQI indexes.

EPA AQI - Found on page 9 of this pdf

Custom ESP Based Sensor

Photo of my DIY AQI and CO2 monitoring device

I also created my own sensor using off the shelf components. You don’t have to use the same exact ESP MCU as me. I initially used an ESP8266, but later iterations of my sensors switched to the ESP32 with Stemma connectors. Stemma is a connector standard that most Adafruit sensors include making it even easier to connect.

Ingredients:

Count	Item	Cost
1	Sensiron SCD-40 CO2 Sensor - There are a few different CO2 sensors with varying accuracy on Adafruit. This CO2 sensor uses some photoacoustic magic to measure how CO2 gas particles exist, whereas some other sensors just estimate the composition	US$58.95
1	Plantower PM2.5 Sensor	US$44.95
1	ESP32-S3 Qt Py w/ Stemma	US$9.95
2	STEMMA QT / Qwiic JST SH 4-pin Cable	US$0.95
	Total	US$115.75

ESPHome

I use ESPHome to build and update the firmware running on the controller. Originally, I wrote the code myself, but ESPHome provides all the glue code needed to pull data from i2c devices and publish it to MQTT.

secrets.yaml

1
2
3
4
5


# Your Wi-Fi SSID and password
wifi_ssid: "SSID"
wifi_password: "PASSWORD"

ota: "RANDOMSTRING"

The following config works with the Adafruit QT Py ESP32-C3 linked earlier. If you have a different board make sure to update the esphome and i2c blocks.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198


esphome:
 name: airquality
 platform: ESP32S3
 board: esp32-s3-devkitc-1

# Enable logging
logger:

ota:
 password: !secret ota

wifi:
 ssid: !secret wifi_ssid
 password: !secret wifi_password

i2c:
 sda: 41
 scl: 40
 scan: false
 frequency: 10kHz

mqtt:
 broker: mqtt.example.com
 discovery_unique_id_generator: mac
 discovery_object_id_generator: device_name
 on_connect:
 - light.turn_on:
 id: neopixel
 effect: connected
 red: 0%
 green: 100%
 blue: 0%
 brightness: 75%
 - delay: 2s
 - light.turn_off:
 id: neopixel
 on_disconnect:
 - light.turn_on:
 id: neopixel
 effect: broken

button:
 - platform: restart
 name: "Restart"

 - platform: template
 name: "CO2 Recalibrate"
 entity_category: config
 on_press:
 then:
 - scd4x.perform_forced_calibration:
 value: !lambda 'return id(co2_cal).state;'

number:
 - platform: template
 name: "CO2 calibration value (ppm)"
 optimistic: true
 min_value: 350
 max_value: 4500
 initial_value: 420
 retain: false
 step: 1
 id: co2_cal
 icon: "mdi:molecule-co2"
 entity_category: "config"

# This board has a small LED that we can use to signal statuses
light:
 - platform: neopixelbus
 type: GRB
 variant: WS2812
 pin: 2
 num_leds: 1
 id: neopixel
 name: "neopixel-enable"
 internal: true
 restore_mode: ALWAYS_OFF
 effects:
 - pulse:
 name: connected
 - strobe:
 name: broken
 colors:
 - state: true
 duration: 500ms
 brightness: 100%
 red: 100%
 green: 0%
 blue: 0%
 - state: true
 brightness: 100%
 duration: 500ms
 red: 100%
 green: 100%
 blue: 0%

sensor:
 - platform: pmsa003i
 pm_1_0:
 name: "PM1.0"
 pm_2_5:
 name: "PM2.5"
 pm_10_0:
 name: "PM10.0"
 pmc_0_3:
 name: "PMC <0.3µm"
 pmc_0_5:
 name: "PMC <0.5µm"
 pmc_1_0:
 name: "PMC <1µm"
 pmc_2_5:
 name: "PMC <2.5µm"
 pmc_5_0:
 name: "PMC <5µm"
 pmc_10_0:
 name: "PMC <10µm"
 update_interval: 60s
 - platform: scd4x
 co2:
 name: "CO2"
 retain: false
 accuracy_decimals: 1
 temperature:
 name: "Temperature"
 accuracy_decimals: 2
 retain: false
 humidity:
 name: "Humidity"
 accuracy_decimals: 1
 retain: false
 # The below number is based on the average air pressure for my
 # area in mBar divided by 1000 (1.009 == 1009mBar)
 # If you don't know what this is, then disable this.
 ambient_pressure_compensation: '1.009'
 # Seems to be broken, see https://github.com/esphome/issues/issues/3063
 temperature_offset: 0
 measurement_mode: low_power_periodic
 # Automatic calibration causes problems when indoors. Instead manually calibrate
 automatic_self_calibration: false
 update_interval: 60s

 # This next block of code converts the PM2.5 measurement into an AQI
 # Based on the US EPA algorithm
 - platform: copy
 source_id: pm25
 id: pm_2_5
 internal: true
 accuracy_decimals: 0
 filters:
 - sliding_window_moving_average:
 window_size: 5
 send_every: 10
 on_value:
 lambda: |-
 // https://en.wikipedia.org/wiki/Air_quality_index#Computing_the_AQI
 if (id(pm_2_5).state < 12.0) {
 // good
 id(aqi_text).publish_state("Good");
 id(pm_2_5_aqi).publish_state((50.0 - 0.0) / (12.0 - 0.0) * (id(pm_2_5).state - 0.0) + 0.0);
 } else if (id(pm_2_5).state < 35.4) {
 // moderate
 id(aqi_text).publish_state("Moderate");
 id(pm_2_5_aqi).publish_state((100.0 - 51.0) / (35.4 - 12.1) * (id(pm_2_5).state - 12.1) + 51.0);
 } else if (id(pm_2_5).state < 55.4) {
 // Unhealthy for Sensitive Groups
 id(aqi_text).publish_state("Unhealthy for Sensitive Groups");
 id(pm_2_5_aqi).publish_state((150.0 - 101.0) / (55.4 - 35.5) * (id(pm_2_5).state - 35.5) + 101.0);
 } else if (id(pm_2_5).state < 150.4) {
 // unhealthy
 id(aqi_text).publish_state("Unhealthy");
 id(pm_2_5_aqi).publish_state((200.0 - 151.0) / (150.4 - 55.5) * (id(pm_2_5).state - 55.5) + 151.0);
 } else if (id(pm_2_5).state < 250.4) {
 // very unhealthy
 id(aqi_text).publish_state("Very Unhealthy");
 id(pm_2_5_aqi).publish_state((300.0 - 201.0) / (250.4 - 150.5) * (id(pm_2_5).state - 150.5) + 201.0);
 } else if (id(pm_2_5).state < 350.4) {
 // hazardous
 id(aqi_text).publish_state("Hazardous");
 id(pm_2_5_aqi).publish_state((400.0 - 301.0) / (350.4 - 250.5) * (id(pm_2_5).state - 250.5) + 301.0);
 } else if (id(pm_2_5).state < 500.4) {
 // hazardous 2
 id(aqi_text).publish_state("Hazardous");
 id(pm_2_5_aqi).publish_state((500.0 - 401.0) / (500.4 - 350.5) * (id(pm_2_5).state - 350.5) + 401.0);
 }

 - platform: template
 name: "PM 2.5 AQI"
 icon: "mdi:air-filter"
 accuracy_decimals: 0
 state_class: measurement
 device_class: aqi
 id: pm_2_5_aqi

text_sensor:
 - platform: template
 name: "AQI Description"
 icon: "mdi:air-filter"
 id: aqi_text

Accuracy and Calibration

One of the issues I’ve noticed with the CO2 sensor is that it seems to show sensor values that are incorrect–below the average outdoor CO2 ppm. The NDIR sensor I have needs to be recalibrated periodically to know how the spectrometer measurements correlate to a specific PPM. The Sensiron SCD-30 data sheet specifies an accuracy of ±30ppm from 400-10k ppm. The auto calibration mode assumes that the sensor is periodically exposed to outdoor fresh air and assumes that outdoor ppm is 400ppm.

ASC assumes that the lowest CO2 concentration the
SCD30 is exposed to corresponds to 400 ppm. The sensor
estimates the most likely reading corresponding to this
background level and identifies this as 400ppm.

SCD32 Field Calibration Guide (pdf)

Due to global CO2 emissions, the average outdoor CO2 concentration is slowly increasing over the years, so this will sensor will slowly diverge:

From Climate.gov. Average CO2 concentration as measured from Mauna Loa Observatory in Hawaii

In addition, the ambient air pressure influences sensor readings and this changes throughout the day due to changing weather conditions.

Thus, we have multiple variables that will all cause drift over time:

Sensor drift due to spectrometer sensor drifts
Global CO2 concentration increasing causing underestimation of CO2 ppm over years
Ambient pressure fluctuations due to changing weather
Not exposing the sensor to fresh air when auto calibration is enabled

There’s only so much I can do to ensure accuracy of this data. I set an average air pressure to compensate and open the window once or twice a week to ensure a reference value.

PurpleAir Indoor Sensor

Before I decided to purchase an air quality sensor, I looked up some independent reviews of the accuracy of different sensors. The US EPA released a report showing that the PurpleAir did reasonably well for the price point, so I bought one to test it out.

Manufacturer Detail Page

Measures:

Air Quality Index
Air Pressure
Temperature
Humidity

PurpleAir sensors can publish data directly to the PurpleAir Map, but I want to export this data to Prometheus or InfluxDB. One way is to use an already built Purple Prometheus Exporter (purpleprom) which fetches from Purple’s API, calculates several AQI metrics, then exposes them to a Prometheus scraper.

This will work, however this depends on their API called over the Internet and only gets the data into Prometheus. My home lab tries to avoid dependencies on Internet services and use local devices where possible.

Luckily, after connecting to my Wi-Fi network, the devices exposes an HTTP endpoint with all sensor data at the URL: http://{ip}:80/json. The following shows an example response (Interesting values are highlighted):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


{
 "SensorId": "01:23:45:67:89:ab",
 "DateTime": "2022/07/26T05:18:38z",
 "Geo": "PurpleAir-1234",
 "Mem": 17552,
 "memfrag": 18,
 "memfb": 14464,
 "memcs": 936,
 "Id": 8436,
 "lat": 47.00000,
 "lon": -122.00000,
 "Adc": 0.02,
 "loggingrate": 15,
 "place": "inside",
 "version": "7.00",
 "uptime": 253112,
 "rssi": -72,
 "period": 119,
 "httpsuccess": 8501,
 "httpsends": 8503,
 "hardwareversion": "2.0",
 "hardwarediscovered": "2.0+BME280+PMSX003-A",
 "current_temp_f": 87,
 "current_humidity": 33,
 "current_dewpoint_f": 54,
 "pressure": 1003.65,
 "p25aqic": "rgb(174,246,0)",
 "pm2.5_aqi": 44,
 "pm1_0_cf_1": 5.4,
 "p_0_3_um": 1215.85,
 "pm2_5_cf_1": 10.65,
 "p_0_5_um": 375.31,
 "pm10_0_cf_1": 10.96,
 "p_1_0_um": 74.28,
 "pm1_0_atm": 5.4,
 "p_2_5_um": 9.75,
 "pm2_5_atm": 10.65,
 "p_5_0_um": 0.58,
 "pm10_0_atm": 10.96,
 "p_10_0_um": 0,
 "pa_latency": 177,
 // ... Some more
}

These fields provide the relevant information to be able to calculate the AQI.

Processing Sensor Data

I now have two different sensors that are ready to provide data. My custom sensor is publishing data through MQTT (and some data is already available in Home Assistant) and the PurpleAir sensor is on the Wi-Fi network and ready. Feel free to skip parts if you have only sensor but not the other.

PurpleAir Sensor

Every minute, my Node Red flow calls the HTTP endpoint on the sensor to grab the latest sensor values, calculates the AQI from the PM2.5 metric, then populates several sensors into Home Assistant.

Here’s the Node Red flow for above. Copy and paste this into the Node Red UI to use it as a template.

Or, using AppDaemon:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


import hassapi as hass
import requests
import aqi

class PurpleAirCollector(hass.Hass):
 def initialize(self):
 self.run_minutely(self.fetch_data, start=None)

 def set_iaqi(self, entity_id, value):
 self.call_service("state/set", entity_id=entity_id, attributes={'state_class': "measurement", 'device_class': 'aqi', 'icon':'mdi:blur', 'friendly_name': 'PurpleAir EPA AQI'}, state=value, namespace="default")

 def fetch_data(self, kwargs):
 resp = requests.get('http://192.168.1.45/json')
 data = resp.json()
 epa = aqi.to_iaqi(aqi.POLLUTANT_PM25, data['pm2_5_cf_1'], algo=aqi.ALGO_EPA)
 self.set_iaqi(entity_id="sensor.living_room_purpleair_1m_iaqi", value=epa)

Custom Sensor

I configured ESPHome to publish raw sensor data to MQTT and HomeAssistant is configured to also connect to the broker and create entities. When I turned on my sensor, I get the following sensors:

The only thing I’m missing is the IAQI metrics. Using the same function node from before, this is easy:

And the JSON for Node Red (copy and paste into the UI):

1

\[{"id":"3f4962eb31025362","type":"tab","label":"Air Quality","disabled":false,"info":"","env":\[\]},{"id":"f12427f26b727dc9","type":"junction","z":"3f4962eb31025362","x":780,"y":380,"wires":\[\["3a856e7ab6d6d4c6","bc26db0709e3d34b"\]\]},{"id":"96894d899bdea100","type":"change","z":"3f4962eb31025362","name":"","rules":\[{"t":"set","p":"sensor\_name","pt":"msg","to":"living\_room","tot":"str"},{"t":"set","p":"pm25","pt":"msg","to":"payload.pm2\_5\_cf\_1","tot":"msg"}\],"action":"","property":"","from":"","to":"","reg":false,"x":640,"y":300,"wires":\[\["f024a8c76b1f9ce0","c121d65dd0f7dc45","f12427f26b727dc9","98a883db43c85b0f"\]\]},{"id":"9e79ed78dd67fc08","type":"switch","z":"3f4962eb31025362","name":"If error","property":"error","propertyType":"msg","rules":\[{"t":"null"},{"t":"nnull"}\],"checkall":"true","repair":false,"outputs":2,"x":470,"y":300,"wires":\[\["96894d899bdea100"\],\[\]\]},{"id":"f024a8c76b1f9ce0","type":"function","z":"3f4962eb31025362","name":"Normalize Temperature","func":"msg.payload = msg.payload.current\_temp\_f - 8;\\n\\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":\[\],"x":890,"y":300,"wires":\[\["9a5bbf8cc529fb67"\]\]},{"id":"3ec53bbca46d0c79","type":"http request","z":"3f4962eb31025362","name":"","method":"GET","ret":"obj","paytoqs":"ignore","url":"http://192.168.1.160/json","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":\[\],"x":310,"y":300,"wires":\[\["9e79ed78dd67fc08"\]\]},{"id":"c121d65dd0f7dc45","type":"switch","z":"3f4962eb31025362","name":"> 0","property":"payload.pressure","propertyType":"msg","rules":\[{"t":"gt","v":"0","vt":"num"}\],"checkall":"true","repair":false,"outputs":1,"x":830,"y":340,"wires":\[\["b1364f17e9a4f113"\]\]},{"id":"b96470e4363c8622","type":"inject","z":"3f4962eb31025362","name":"every minute","props":\[{"p":"payload"}\],"repeat":"60","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":300,"wires":\[\["3ec53bbca46d0c79"\]\]},{"id":"98a883db43c85b0f","type":"link call","z":"3f4962eb31025362","name":"","links":\["4e1646049cd8c3fa"\],"linkType":"static","timeout":"30","x":860,"y":260,"wires":\[\["fae3173cf70b26fc"\]\]},{"id":"3a856e7ab6d6d4c6","type":"ha-sensor","z":"3f4962eb31025362","name":"PurpleAir Humidity","entityConfig":"feea3b65bc43902a","version":0,"state":"payload.current\_humidity","stateType":"msg","attributes":\[\],"inputOverride":"allow","outputProperties":\[\],"x":1150,"y":420,"wires":\[\[\]\]},{"id":"9a5bbf8cc529fb67","type":"ha-sensor","z":"3f4962eb31025362","name":"PurpleAir Temperature","entityConfig":"24eaa8400b7264e6","version":0,"state":"payload","stateType":"msg","attributes":\[\],"inputOverride":"allow","outputProperties":\[\],"x":1160,"y":300,"wires":\[\[\]\]},{"id":"fae3173cf70b26fc","type":"ha-sensor","z":"3f4962eb31025362","name":"PurpleAir 1m EPA","entityConfig":"5bd2e57458ace436","version":0,"state":"iaqi.epa","stateType":"msg","attributes":\[{"property":"type","value":"iaqi","valueType":"str"}\],"inputOverride":"allow","outputProperties":\[\],"x":1150,"y":240,"wires":\[\[\]\]},{"id":"b1364f17e9a4f113","type":"ha-sensor","z":"3f4962eb31025362","name":"PurpleAir Air Pressure","entityConfig":"3d98f8971f231416","version":0,"state":"payload.pressure","stateType":"msg","attributes":\[\],"inputOverride":"allow","outputProperties":\[\],"x":1160,"y":360,"wires":\[\[\]\]},{"id":"bc26db0709e3d34b","type":"ha-sensor","z":"3f4962eb31025362","name":"PurpleAir Wi-Fi rx RSSI","entityConfig":"d04ff191e844b1a0","version":0,"state":"payload.rssi","stateType":"msg","attributes":\[\],"inputOverride":"allow","outputProperties":\[\],"x":1170,"y":480,"wires":\[\[\]\]},{"id":"4e1646049cd8c3fa","type":"link in","z":"3f4962eb31025362","name":"Compute AQI","links":\[\],"x":265,"y":500,"wires":\[\["c256eaec2e31c57f"\]\]},{"id":"bb93a33f9c5e9a7d","type":"link out","z":"3f4962eb31025362","name":"link out 2","mode":"return","links":\[\],"x":575,"y":500,"wires":\[\]},{"id":"c256eaec2e31c57f","type":"function","z":"3f4962eb31025362","name":"Compute AQI","func":"// https://github.com/hrbonz/python-aqi/blob/master/aqi/algos/epa.py\\nconst data = {\\n 'aqi': \[\\n \[0, 50\],\\n \[51, 100\],\\n \[101, 150\],\\n \[151, 200\],\\n \[201, 300\],\\n \[301, 400\],\\n \[401, 500\]\\n \],\\n 'bp': {\\n POLLUTANT\_PM25: \[\\n \[0.0, 12.0\],\\n \[12.1, 35.4\],\\n \[35.5, 55.4\],\\n \[55.5, 150.4\],\\n \[150.5, 250.4\],\\n \[250.5, 350.4\],\\n \[350.5, 500.4\],\\n \]\\n }\\n}\\n\\nfunction convertToIaqi(value) {\\n if (value < 0) {\\n throw new Error(\`PM2.5 can't be negative: ${value}\`);\\n }\\n const adjValue = Math.floor(value \* 10) / 10;\\n const breakpoints = data.bp.POLLUTANT\_PM25;\\n var bplo = 0;\\n var bphi = 0;\\n var index = 0;\\n for (const bp of breakpoints) {\\n if (adjValue >= bp\[0\] && adjValue <= bp\[1\]) {\\n bplo = bp\[0\];\\n bphi = bp\[1\];\\n break;\\n }\\n index++;\\n }\\n \\n const aqival = data.aqi\[index\];\\n if (!aqival) {\\n throw new Error(\`Can't calculate AQI for '${value}' index out of range: ${index}.\`);\\n }\\n //return index;\\n const aqiValue = (aqival\[1\] - aqival\[0\]) / (bphi - bplo) \* (adjValue - bplo) + aqival\[0\];\\n \\n return Math.round(aqiValue);\\n}\\n\\nmsg.iaqi = {\\n epa: convertToIaqi(msg.pm25),\\n aqandu: convertToIaqi(0.778 \* msg.pm25 + 2.65),\\n lrapa: convertToIaqi(Math.max(0, 0.5 \* msg.pm25 - 0.66))\\n};\\n\\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":\[\],"x":420,"y":500,"wires":\[\["bb93a33f9c5e9a7d"\]\]},{"id":"feea3b65bc43902a","type":"ha-entity-config","server":"434f6479916be393","deviceConfig":"","name":"sensor config for PurpleAir Humidity","version":"6","entityType":"sensor","haConfig":\[{"property":"name","value":"Humidity"},{"property":"icon","value":"mdi:water-percent"},{"property":"entity\_category","value":""},{"property":"device\_class","value":"humidity"},{"property":"unit\_of\_measurement","value":"%"},{"property":"state\_class","value":"measurement"}\],"resend":true},{"id":"24eaa8400b7264e6","type":"ha-entity-config","server":"434f6479916be393","deviceConfig":"","name":"sensor config for PurpleAir Temperature","version":"6","entityType":"sensor","haConfig":\[{"property":"name","value":"Temperature"},{"property":"icon","value":""},{"property":"entity\_category","value":""},{"property":"device\_class","value":"temperature"},{"property":"unit\_of\_measurement","value":"°F"},{"property":"state\_class","value":"measurement"}\],"resend":true},{"id":"5bd2e57458ace436","type":"ha-entity-config","server":"434f6479916be393","deviceConfig":"","name":"sensor config for PurpleAir 1m EPA","version":"6","entityType":"sensor","haConfig":\[{"property":"name","value":"AQI (EPA)"},{"property":"icon","value":"mdi:blur"},{"property":"entity\_category","value":""},{"property":"device\_class","value":"aqi"},{"property":"unit\_of\_measurement","value":"aqi"},{"property":"state\_class","value":"measurement"}\],"resend":true},{"id":"3d98f8971f231416","type":"ha-entity-config","server":"434f6479916be393","deviceConfig":"","name":"sensor config for PurpleAir Air Pressure","version":"6","entityType":"sensor","haConfig":\[{"property":"name","value":"Air Pressure"},{"property":"icon","value":"mdi:gauge"},{"property":"entity\_category","value":""},{"property":"device\_class","value":"pressure"},{"property":"unit\_of\_measurement","value":"mbar"},{"property":"state\_class","value":"measurement"}\],"resend":true},{"id":"d04ff191e844b1a0","type":"ha-entity-config","server":"434f6479916be393","deviceConfig":"","name":"PurpleAir Wi-Fi RSSI","version":"6","entityType":"sensor","haConfig":\[{"property":"name","value":"Wi-Fi RSSI"},{"property":"icon","value":""},{"property":"entity\_category","value":"diagnostic"},{"property":"device\_class","value":"signal\_strength"},{"property":"unit\_of\_measurement","value":"dBm"},{"property":"state\_class","value":"measurement"}\],"resend":false},{"id":"434f6479916be393","type":"server","name":"Home Assistant","version":5,"addon":false,"rejectUnauthorizedCerts":true,"ha\_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true,"heartbeat":false,"heartbeatInterval":"30","areaSelector":"friendlyName","deviceSelector":"friendlyName","entitySelector":"friendlyName","statusSeparator":"at: ","statusYear":"hidden","statusMonth":"short","statusDay":"numeric","statusHourCycle":"h23","statusTimeFormat":"h:m","enableGlobalContextStore":true}\]

Publishing to InfluxDB

HomeAssistant stores sensor data in the Recorder, which by default uses SQLite, but can be changed to use any SQL database like Postgres or MySQL. These aren’t efficient at storing large amounts of time series data, so instead I store time series data long-term in InfluxDB using the HA Integration.

HomeAssistant by default publishes all entities and all attributes for each state to InfluxDB. This results in *a lot* of useless data being stored and will quickly consume a large amount of disk space. To prevent this, I will explicitly whitelist just the sensor data and ignore any attributes that aren’t useful.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


influxdb:
 host: influxdb-influxdb2.datastore.svc.cluster.local.
 port: 8086
 ssl: false
 api_version: 2
 token: 'token'
 bucket: homeassistant
 organization: influxdata
 component_config_domain:
 sensor:
 ignore_attributes:
 - attribution
 - device_class
 - state_class
 - last_reset
 - integration
 - description
 - unit_of_measurement
 - type
 include:
 domain:
 - sensor

Grafana

Grafana provides the ability to build richer dashboards compared to HomeAssistant. I created an Air Quality monitoring dashboard that fetches data from InfluxDB:

Grafana Dashboard JSON. To import, go to Grafana > Dashboards > Import.

1

{"\_\_inputs":\[{"name":"DS\_INFLUXDB","label":"InfluxDB","description":"","type":"datasource","pluginId":"influxdb","pluginName":"InfluxDB"}\],"\_\_elements":{},"\_\_requires":\[{"type":"panel","id":"gauge","name":"Gauge","version":""},{"type":"grafana","id":"grafana","name":"Grafana","version":"9.0.5"},{"type":"datasource","id":"influxdb","name":"InfluxDB","version":"1.0.0"},{"type":"panel","id":"state-timeline","name":"State timeline","version":""},{"type":"panel","id":"timeseries","name":"Time series","version":""}\],"annotations":{"list":\[{"builtIn":1,"datasource":{"type":"datasource","uid":"grafana"},"enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","target":{"limit":100,"matchAny":false,"tags":\[\],"type":"dashboard"},"type":"dashboard"}\]},"editable":true,"fiscalYearStartMonth":0,"graphTooltip":0,"id":null,"links":\[\],"liveNow":false,"panels":\[{"collapsed":false,"datasource":{"type":"prometheus","uid":"PBFA97CFB590B2093"},"gridPos":{"h":1,"w":24,"x":0,"y":0},"id":18,"panels":\[\],"title":"Current AQI","type":"row"},{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":\[\],"max":500,"min":0,"thresholds":{"mode":"absolute","steps":\[{"color":"#009966","value":null},{"color":"#ffde33","value":50},{"color":"#ff9933","value":100},{"color":"#cc0033","value":150},{"color":"#660099","value":200},{"color":"#7e0023","value":300}\]}},"overrides":\[\]},"gridPos":{"h":5,"w":3,"x":0,"y":1},"id":19,"options":{"orientation":"auto","reduceOptions":{"calcs":\["lastNotNull"\],"fields":"","values":false},"showThresholdLabels":false,"showThresholdMarkers":true,"text":{}},"pluginVersion":"9.0.5","targets":\[{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"query":"import \\"strings\\"\\r\\n\\r\\nfrom(bucket: \\"homeassistant\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r\[\\"\_measurement\\"\] == \\"aqi\\")\\r\\n |> filter(fn: (r) => r\[\\"\_field\\"\] == \\"value\\")\\r\\n |> filter(fn: (r) => strings.hasSuffix(v: r\[\\"entity\_id\\"\], suffix: \\"\_iaqi\\"))\\r\\n |> last()\\r\\n |> keep(columns: \[\\"\_value\\"\])\\r\\n |> mean()","refId":"A"}\],"title":"Current PM 2.5 AQI","type":"gauge"},{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"fieldConfig":{"defaults":{"mappings":\[\],"max":500,"min":0,"thresholds":{"mode":"absolute","steps":\[{"color":"#009966","value":null},{"color":"#ffde33","value":50},{"color":"#ff9933","value":100},{"color":"#cc0033","value":150},{"color":"#660099","value":200},{"color":"#7e0023","value":300}\]}},"overrides":\[\]},"gridPos":{"h":5,"w":3,"x":3,"y":1},"id":20,"options":{"orientation":"auto","reduceOptions":{"calcs":\["mean"\],"fields":"","values":false},"showThresholdLabels":false,"showThresholdMarkers":true,"text":{}},"pluginVersion":"9.0.5","targets":\[{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"query":"import \\"strings\\"\\r\\n\\r\\nfrom(bucket: \\"homeassistant\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r\[\\"\_measurement\\"\] == \\"aqi\\")\\r\\n |> filter(fn: (r) => r\[\\"\_field\\"\] == \\"value\\")\\r\\n |> filter(fn: (r) => strings.hasSuffix(v: r\[\\"entity\_id\\"\], suffix: \\"\_aqandu\\"))\\r\\n |> last()\\r\\n |> keep(columns: \[\\"\_value\\"\])\\r\\n |> mean()","refId":"A"}\],"title":"Current PM 2.5 AQI (AQandU)","type":"gauge"},{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"fieldConfig":{"defaults":{"mappings":\[\],"max":500,"min":0,"thresholds":{"mode":"absolute","steps":\[{"color":"#009966","value":null},{"color":"#ffde33","value":50},{"color":"#ff9933","value":100},{"color":"#cc0033","value":150},{"color":"#660099","value":200},{"color":"#7e0023","value":300}\]}},"overrides":\[\]},"gridPos":{"h":5,"w":3,"x":6,"y":1},"id":21,"options":{"orientation":"auto","reduceOptions":{"calcs":\["mean"\],"fields":"","values":false},"showThresholdLabels":false,"showThresholdMarkers":true,"text":{}},"pluginVersion":"9.0.5","targets":\[{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"query":"import \\"strings\\"\\r\\n\\r\\nfrom(bucket: \\"homeassistant\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r\[\\"\_measurement\\"\] == \\"aqi\\")\\r\\n |> filter(fn: (r) => r\[\\"\_field\\"\] == \\"value\\")\\r\\n |> filter(fn: (r) => strings.hasSuffix(v: r\[\\"entity\_id\\"\], suffix: \\"\_lrapa\\"))\\r\\n |> last()\\r\\n |> keep(columns: \[\\"\_value\\"\])\\r\\n |> mean()","refId":"A"}\],"title":"Current PM 2.5 AQI (LRAPA)","type":"gauge"},{"collapsed":false,"datasource":{"type":"prometheus","uid":"PBFA97CFB590B2093"},"gridPos":{"h":1,"w":24,"x":0,"y":6},"id":8,"panels":\[\],"title":"AQI History","type":"row"},{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"area"}},"mappings":\[\],"thresholds":{"mode":"absolute","steps":\[{"color":"#009966","value":null},{"color":"#ffde33","value":50},{"color":"#ff9933","value":100},{"color":"#cc0033","value":150},{"color":"#660099","value":200},{"color":"#7e0023","value":300}\]}},"overrides":\[{"matcher":{"id":"byFrameRefID","options":"G"},"properties":\[{"id":"color","value":{"fixedColor":"blue","mode":"fixed"}}\]},{"matcher":{"id":"byFrameRefID","options":"F"},"properties":\[{"id":"color","value":{"fixedColor":"green","mode":"fixed"}}\]},{"matcher":{"id":"byFrameRefID","options":"E"},"properties":\[{"id":"color","value":{"fixedColor":"red","mode":"fixed"}}\]}\]},"gridPos":{"h":11,"w":8,"x":0,"y":7},"id":2,"options":{"legend":{"calcs":\[\],"displayMode":"list","placement":"bottom"},"tooltip":{"mode":"single","sort":"none"}},"pluginVersion":"8.3.3","targets":\[{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"query":"from(bucket: \\"homeassistant\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r\[\\"\_measurement\\"\] == \\"aqi\\")\\r\\n |> filter(fn: (r) => r\[\\"\_field\\"\] == \\"value\\")\\r\\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\\r\\n |> yield(name: \\"mean\\")","refId":"A"}\],"title":"PM 2.5 AQI","type":"timeseries"},{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":\[\],"max":100,"min":0,"thresholds":{"mode":"absolute","steps":\[{"color":"green","value":null}\]},"unit":"percent"},"overrides":\[\]},"gridPos":{"h":11,"w":7,"x":8,"y":7},"id":23,"options":{"legend":{"calcs":\[\],"displayMode":"list","placement":"bottom"},"tooltip":{"mode":"single","sort":"none"}},"targets":\[{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"query":"from(bucket: \\"homeassistant\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r\[\\"\_measurement\\"\] == \\"%\\")\\r\\n |> filter(fn: (r) => r\[\\"\_field\\"\] == \\"value\\")\\r\\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\\r\\n |> yield(name: \\"mean\\")","refId":"A"}\],"title":"Humidity","type":"timeseries"},{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisLabel":"°F","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"line"}},"mappings":\[\],"thresholds":{"mode":"absolute","steps":\[{"color":"yellow","value":null},{"color":"green","value":60},{"color":"red","value":80}\]}},"overrides":\[{"matcher":{"id":"byFrameRefID","options":"A"},"properties":\[{"id":"displayName","value":"${\_\_field.labels.entity\_id}"}\]},{"matcher":{"id":"byFrameRefID","options":"B"},"properties":\[{"id":"color","value":{"mode":"fixed"}}\]}\]},"gridPos":{"h":11,"w":9,"x":15,"y":7},"id":39,"interval":"5m","options":{"legend":{"calcs":\[\],"displayMode":"list","placement":"bottom"},"tooltip":{"mode":"single","sort":"none"}},"targets":\[{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"query":"import \\"strings\\"\\r\\n\\r\\nfrom(bucket: \\"homeassistant\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r\[\\"entity\_id\\"\] != \\"fridge\_door\_temperature\_measurement\\" and not strings.hasSuffix(v: r\[\\"entity\_id\\"\], suffix: \\"index\\") and not strings.hasSuffix(v: r\[\\"entity\_id\\"\], suffix: \\"point\\"))\\r\\n |> filter(fn: (r) => r\[\\"\_measurement\\"\] == \\"°F\\")\\r\\n |> filter(fn: (r) => r\[\\"\_field\\"\] == \\"value\\")\\r\\n |> filter(fn: (r) => r\[\\"domain\\"\] == \\"sensor\\")\\r\\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: true)\\r\\n |> fill(usePrevious: true)\\r\\n |> map(fn: (r) => ({ r with name: r.entity\_id }))\\r\\n |> yield(name: \\"mean\\")\\r\\n\\r\\n// |> filter(fn: (r) => r\[\\"entity\_id\\"\] == \\"living\_room\_temperature\\" or r\[\\"entity\_id\\"\] == \\"home\_temperature\\" or r\[\\"entity\_id\\"\] == \\"button\_temperature\_measurement\\" or r\[\\"entity\_id\\"\] == \\"bedroom\_temperature\_esp\\" or r\[\\"entity\_id\\"\] == \\"bedroom\_temperature\_ecobee\\" or r\[\\"entity\_id\\"\] == \\"hallway\_button\_temperature\_measurement\\")","refId":"A"}\],"title":"Temperature","type":"timeseries"},{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"line"}},"decimals":4,"mappings":\[\],"thresholds":{"mode":"absolute","steps":\[{"color":"green","value":null}\]},"unit":"pressurembar"},"overrides":\[\]},"gridPos":{"h":8,"w":3,"x":0,"y":18},"id":33,"options":{"legend":{"calcs":\[\],"displayMode":"list","placement":"bottom"},"tooltip":{"mode":"single","sort":"none"}},"targets":\[{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"query":"from(bucket: \\"homeassistant\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r\[\\"\_measurement\\"\] == \\"mbar\\")\\r\\n |> filter(fn: (r) => r\[\\"\_field\\"\] == \\"value\\")\\r\\n |> filter(fn: (r) => r\[\\"domain\\"\] == \\"sensor\\")\\r\\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\\r\\n |> yield(name: \\"mean\\")","refId":"A"}\],"title":"Air Pressure","type":"timeseries"},{"alert":{"alertRuleTags":{},"conditions":\[{"evaluator":{"params":\[1000\],"type":"gt"},"operator":{"type":"and"},"query":{"params":\["A","5m","now"\]},"reducer":{"params":\[\],"type":"avg"},"type":"query"}\],"executionErrorState":"alerting","for":"5m","frequency":"1m","handler":1,"name":"CO2 PPM alert","noDataState":"no\_data","notifications":\[\]},"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"custom":{"axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"area"}},"links":\[\],"mappings":\[\],"thresholds":{"mode":"absolute","steps":\[{"color":"green","value":null},{"color":"#EAB839","value":900},{"color":"red","value":1000}\]},"unit":"ppm"},"overrides":\[\]},"gridPos":{"h":7,"w":4,"x":3,"y":18},"id":29,"options":{"legend":{"calcs":\[\],"displayMode":"list","placement":"bottom"},"tooltip":{"mode":"single","sort":"none"}},"targets":\[{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"query":"from(bucket: \\"homeassistant\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r\[\\"\_measurement\\"\] == \\"ppm\\")\\r\\n |> filter(fn: (r) => r\[\\"\_field\\"\] == \\"value\\")\\r\\n |> filter(fn: (r) => r\[\\"domain\\"\] == \\"sensor\\")\\r\\n |> filter(fn: (r) => r\[\\"entity\_id\\"\] == \\"bedroom\_co2\\")\\r\\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\\r\\n |> yield(name: \\"mean\\")","refId":"A"}\],"thresholds":\[{"colorMode":"critical","op":"gt","value":1000,"visible":true}\],"title":"CO2 PPM","type":"timeseries"},{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"custom":{"fillOpacity":70,"lineWidth":0,"spanNulls":false},"displayName":"${\_\_field.labels.entity\_id}","mappings":\[\],"thresholds":{"mode":"absolute","steps":\[{"color":"#009966","value":null},{"color":"#ffde33","value":50},{"color":"#ff9933","value":100},{"color":"#cc0033","value":150},{"color":"#660099","value":200},{"color":"#7e0023","value":300}\]}},"overrides":\[\]},"gridPos":{"h":7,"w":10,"x":7,"y":18},"id":31,"options":{"alignValue":"left","legend":{"displayMode":"list","placement":"bottom"},"mergeValues":true,"rowHeight":0.9,"showValue":"never","tooltip":{"mode":"single","sort":"none"}},"targets":\[{"datasource":{"type":"influxdb","uid":"${DS\_INFLUXDB}"},"query":"import \\"strings\\"\\r\\n\\r\\nfrom(bucket: \\"homeassistant\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r\[\\"\_measurement\\"\] == \\"aqi\\")\\r\\n |> filter(fn: (r) => r\[\\"\_field\\"\] == \\"value\\")\\r\\n |> filter(fn: (r) => strings.hasSuffix(v: r\[\\"entity\_id\\"\], suffix: \\"\_iaqi\\"))\\r\\n |> yield(name: \\"mean\\")","refId":"A"}\],"title":"Air Quality","type":"state-timeline"}\],"refresh":"","schemaVersion":36,"style":"dark","tags":\[\],"templating":{"list":\[\]},"time":{"from":"now-3h","to":"now"},"timepicker":{"refresh\_intervals":\["10s","30s","1m","5m","15m","30m","1h","2h","1d"\]},"timezone":"America/Los\_Angeles","title":"Air Quality (Public)","uid":"ASFASr23rfasdf","version":4,"weekStart":""}

Conclusion

In this post, I walked through two different sensors: a DIY sensor and a prebuilt solution. Showed how to connect to Home Assistant, InfluxDB, and create some basic dashboards for visualization. Future projects include improving accuracy or even designing a custom PCB instead of stringing together sensors with wires.

Updates

Jan 2023

After running this for a few months, I noticed that my measured CO2 started getting less accurate. It seemed to get in a bad state until I reset the esp8266, then it would work okay.

A detailed chart showing inaccurate readings

A long-term view

There were several relevant looking GitHub issues all reporting similar behavior:

To fix this issue, I temporarily set temperature_offset to be 0 and redeployed.

July 2023

Occasionally after rebooting the esp device, one of the two sensors would just not come online and report data. It was hard to track down, but I believe that setting i2c.scan: false made it more reliable.

September 2023

Changed to the Adafruit Qt Py ESP32-S3 which includes a stemma connector
Updated the ESPHome YAML to match the board and pinout
Added AQI calculation on device so we don’t need the Node Red flow for the custom sensor

Accurate, Local Home Energy Monitoring: Part 3 – Software Config

Sun, 14 Aug 2022 00:00:00 +0000

In the previous post in this series, I selected an energy monitoring system that is purely local based (no cloud), integrates into the breaker box, and showed how to connect it to the network and configure the size of each circuit. In this post, I’ll show how to connect the BrulTech GreenEye Energy Monitor to HomeAssistant and create some useful monitoring dashboards.

GreenEye Monitor Firmware

While trying to connect my monitor to Home Assistant, I came across a firmware bug in the GreenEye Monitor and found a forum thread that Brultech had a bug with their packet formats which has been fixed in firmware version 5.39+. To check and find the serial number which will be needed, navigate to http://{monitor ip}:8000/, then click “Enter Setup Mode”.

Make note of the serial number and firmware version in the top right corner.

If you’re not on 5.39 or above, download the latest version here and the network utility here. Connect to the monitor, then update the firmware.

On the top, blue menu panel click “Packet Send”, then change both COM1 and COM2 Packet Formats to Bin32 NET, then click save.

Home Assistant Setup

The integration for the GreenEye Monitor in HA is a bit out of date and can’t be easily added via the UI like some other integrations, however several developers are working on improving the state and moving to HACS. I’ll be following along and contributing now that I’ve got a working setup. Until that’s available, let’s install with the existing integration.

Open up the Home Assistant configuration.yml file, and add the following. Provide a short and meaningful name or even just use what’s labeled on the circuit breaker. I used the common prefix “main_panel_”, so I can create templates that reference all power sources on this breaker.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


greeneye_monitor:
 port: 8000
 monitors:
 - serial_number: "01234567"
 channels:
 - number: 1
 name: main_panel_kitchen
 - number: 2
 name: main_panel_bathroom
 # For each channel
 voltage:
 - number: 1
 name: main_panel_volts

Save the file, then restart Home Assistant for the changes to take affect.

Kubernetes Networking

The GreenEye needs to open a TCP connection to Home Assistant, however this is complicated in my setup because I’m using Kubernetes and all traffic to HA goes through my ingress controller, ingress-nginx. If you’re not doing this, skip ahead to the next step.

The issue is that NGINX is only proxying HTTP/HTTPS traffic.

While it’s possible to use NGINX to proxy TCP ports, I wanted to preserve the source port which can’t be done with this TCP protocol. The easiest thing is to create a separate Layer 4 Load Balancer to forward just the port we need:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


apiVersion: v1
kind: Service
metadata:
 name: homeassistant
spec:
 allocateLoadBalancerNodePorts: false
 externalTrafficPolicy: Local
 ports:
 - name: greeneye
 port: 8000
 protocol: TCP
 targetPort: 8000
 selector:
 workload.user.cattle.io/workloadselector: apps.deployment-smarthome-homeassistant
 sessionAffinity: None
 type: LoadBalancer
status:
 loadBalancer:
 ingress:
 - ip: 192.168.6.3

Once the load balancer is created, make note of the IP address in the status block. This will be needed in the next step.

Start sending power data

Then switch to the Network page. Enter the IP address of the load balancer created above or the IP address for Home Assistant in the box, then click save

After a few seconds, Home Assistant should start populating the dashboard with live power data.

Setting up the HA dashboard

Required Attributes

This integration does not correctly set the attributes so that Home Assistant treats this data as power and instead treats it as opaque sensor data. The only way to fix this is to create a *new* template sensor with the correct attributes. In my case, I didn’t have a CT covering total usage for my main panel, so I summed everything together:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


template:
 - sensor:
 - name: Main Panel Total Power
 device_class: power
 state_class: measurement
 unit_of_measurement: W
 state: >
 {{ states.sensor
 | selectattr('entity_id', 'match', 'sensor.main_panel_*')
 | selectattr('attributes.unit_of_measurement', 'equalto', 'W')
 | rejectattr('state', 'equalto', 'unknown')
 | rejectattr('state', 'equalto', 'unavailable')
 | rejectattr('entity_id', 'equalto', 'sensor.main_panel_total_power')
 | map(attribute='state')
 | map('float')
 | sum }}

However, if you do have a total CT, then you can just override the attributes:

1
2
3
4
5
6


homeassistant:
 customize:
 sensor.main_panel_total_power:
 device_class: power
 state_class: measurement
 unit_of_measurement: W

Watts to kilowatt-hours

The energy monitor emits volts and watts, but this isn’t very useful because watts is instantaneous power usage, but electric companies and the HA energy dashboard both need to work with kWh (kilowatt-hours, 1kWh = 1,000 watts of power for 1 hour.) To convert from Watts to kWh, we need to integrate the value over time using a Riemann sum. This isn’t perfectly accurate, but since the input data is discretized over 5 second intervals, it’s close enough.

In Home Assistant, go to Settings > Integrations > Helpers.

Click Create helper > Integration - Riemann sum integral sensor

Name your sensor something like “Main Panel Total Energy” (Energy because Energy is total power over time.
Use the Left Riemann sum method (why?)
Pick the kilo metric prefix

Then click submit. Give it a minute or two to update.

Knowing the current kWh rate

A lot of electricity providers have different rates through the year or even consumption based changes that increase the rate as you use more per day or month. My provider, Seattle City Light, publishes their rates here. They have summer rates and winter rates.

	USD/kWh
Base service charge per day (Ignored for calculations)	$0.1974
1st block per kWh (Summer is >10kWh and Winter is >16kWh)	$0.1056
2nd block per kWh	$0.1307

First, I need to create a helper that tracks my daily usage and resets every night. Create a helper in the Home Assistant UI in Settings > Devices & Services > Helpers.

Create a Utility Meter helper and enter a name and the input sensor. The name is used to create an entity name so it must match what you specify in the template in the next step.

Next, I defined a new template sensor. The following sensor is based on my rates, but you can adjust as you need.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


template:
 - sensor:
 - name: Electricity kWh Rate
 unit_of_measurement: USD/kWh
 state: >-
 {% set month = now().month %}
 {% set isSummer = month >= 4 and month <= 9 %}
 {% set buckets = [0.1056, 0.1307] %}
 {% set bucketskWh = {
 'winter': 16,
 'summer': 10
 }
 %}
 {% set seasonalThreshold = bucketskWh['summer' if isSummer else 'winter'] %}
 {% if states('sensor.daily_electricity_usage') | float > seasonalThreshold %}
 {{ buckets[1] }}
 {% else %}
 {{ buckets[0] }}
 {%- endif %}

After that reload the templates in Developer Tools > YAML > YAML Reload Template Sensors.

Setup the Energy Dashboard

Now you should be able to go to the Energy Dashboard and select the newly created Energy sensor for Grid consumption and rate sensor.

After a few hours you should start seeing metrics in this dashboard.

Reducing Recorder Usage

By default, the GreenEye Monitor sends updated data every 5 seconds. This ensures good accuracy, but produces too much data that isn’t useful for long-term statistics.

1
2
3
4


recorder:
 exclude:
 entities:
 - sensor.main_panel_*_power

Conclusion

In this post, I walked through how to connect the BrulTech to Home Assistant. In the next post, I’ll walk through how to manage data sizes with downsampling and data retention policies.

The Brultech is definitely not as polished as some of the other options, but it was very powerful and provided rich data without depending on the cloud.

Errata

2022-11-29 - The template to aggregate up the total power did not ignore irrelevant sensors, such as those named sensors.main_panel_total_energy, which caused the sum to infinitely increase.

Split Horizon DNS with external-dns and cert-manager for Kubernetes

Tue, 21 Jun 2022 00:00:00 +0000

There were a few services that I ran that I wanted to be able to access from both inside my home network and outside my home network. If I was inside my home network, I wanted to route directly to the service, but if I was outside I needed to be able to route traffic through a proxy that would then route into my home lab. Additionally, I wanted to support SSL on all my services for security using cert-manager

Since my IPv4 addresses differ inside my network vs outside, I need to use split-horizon DNS to respond with the correct DNS query. Split-horizon DNS refers to the DNS on one horizon (inside the network) showing different results than outside the network.

One day, we’ll all be on IPv6 and this won’t be needed anymore because all services will be using globally unique IPv6 addresses, but a las we’re stuck in IPv4 land. With IPv6, instead of having separate views of the IP addresses, all services could just get a single global IPv6 address registered. Then no matter where you are, the same IPv6 address gets routed to the correct Kubernetes service.

My cluster is using MetalLB as it’s L4 load balancer and it or an equivalent is required to be able to forward IP-level packets to the DNS server. NodePorts don’t work because they use a random port and other servers expect to be able to use port 53/udp and 53/tcp.

Prior to this, I purchased a domain name, mydomain.com. I reserved a subdomain, *.home.mydomain.com that all of my home lab software will run on-top of. I usually purchase my domain names from porkbun.com, but any domain name registrar will work.

The Problem with TLS

To provision certificates, Let’s Encrypt needs to confirm that you own or at least have control over the DNS domain name that you’re requesting. Since I’m running home lab inside my private network that’s not accessible to the internet, it won’t be able to verify I own *.home.mydomain.com. To fix this, I need cert-manager to be able to create a DNS record on a publicly resolvable DNS record.

The diagram below shows the problem with split horizon DNS in this case. cert-manager needs to be able to update the external DNS server and be able to verify the DNS record is updated, but if it hits the internal DNS server then it won’t be able to confirm everything is working.

Architectural diagram showing how DNS queries progress through the network. Clients call into Pi-Hole, which then conditionally forwards the lab zone to the CoreDNS instance, everything else goes to the internet. CoreDNS then queries etcd for the lab zone results.

Solution

Let’s break down the different components and how to configure each one.

Component diagram showing the software we’re going to deploy

Ingress-NGINX

Ingress controllers in Kubernetes automatically update each Ingress resource’s status block with an IP address that can be used to access that Ingress. However, there are multiple IP addresses that can be associated with a given ingress. If you’re using a NodePort service, then that ingress would be pointing to the Node’s IP, but a Layer 3 LB service, would need to use the IP of the ingress-nginx’s LB.

See the below example, this ingress is pointing to the node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: sonos-api
spec:
 # ...
status:
 loadBalancer:
 ingress:
 - ip: 192.168.2.196 # This is the node's IP, not the service IP

The correct IP as exposed by MetalLB for NGINX is 192.168.6.8. To fix this, we need to tell ingress-nginx to instead use it’s service IP. If you’ve deployed nginx using Helm, then change the values.yaml

1
2
3


controller:
 publishService:
 enabled: true # Change from false to true

After deploying, NGINX should modify the Ingress statuses to point to the correct IP:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: sonos-api
spec:
 # ...
status:
 loadBalancer:
 ingress:
 - ip: **192.168.6.8**

etcd

External-dns will store the DNS records in etcd and CoreDNS will look up query responses for the zone in in etcd.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: etcd
 namespace: dns
spec:
 selector:
 matchLabels:
 workload.user.cattle.io/workloadselector: apps.statefulset-dns-etcd
 template:
 metadata:
 labels:
 workload.user.cattle.io/workloadselector: apps.statefulset-dns-etcd
 spec:
 containers:
 - command:
 - /bin/sh
 - '-c'
 - |
 exec etcd --name ${HOSTNAME} \\
 --listen-peer-urls http://0.0.0.0:2380 \\
 --listen-client-urls http://0.0.0.0:2379 \\
 --advertise-client-urls http://${HOSTNAME}.etcd:2379 \\
 --initial-advertise-peer-urls http://${HOSTNAME}:2380 \\
 --initial-cluster-token etcd-cluster-1 \\
 --initial-cluster-state new \\
 --data-dir /var/run/etcd/default.etcd
 image: quay.io/coreos/etcd:latest
 name: etcd
 ports:
 - containerPort: 2379
 name: client
 protocol: TCP
 - containerPort: 2380
 name: peer
 protocol: TCP
 volumeMounts:
 - mountPath: /var/run/etcd
 name: data
 volumes:
 - hostPath:
 path: /home/docker/dns-etcd
 type: ''
 name: data
 replicas: 1

External-DNS

External-DNS is a Kubernetes tool that will take ingresses or services defined in Kubernetes and automatically create DNS records in some provider for you. For ingresses, it takes the IP address defined in the status of the Ingress by the ingress controller (ingress-nginx.)

Following the Helm install method, udate the following Helm values. Note that I’m using the K8s namespace ‘dns’. If you deploy your etcd/external-dns in a different namespace, make sure to update ETCD_URLs below. Also note, I’m using the fully qualified names because of an issue I discovered and documented in this post.

1
2
3
4
5
6
7
8


env:
 - name: ETCD_URLS
 value: http://etcd.dns.svc.cluster.local.:2379

provider: coredns
sources:
 - ingress
 - service

Once deployed, check the logs to verify that the records are being created correctly.

CoreDNS

CoreDNS will answer the DNS queries by querying etcd for records.

Kubernetes has a limitation where we can’t use TCP and UDP on the same Load Balancer service.

To fix this, you have two options:

Enable the feature gate MixedProtocolLBService
Disable TCP based DNS queries and hope you don’t exceed the size of a UDP packet (ref.)

Many Kubernetes clusters come built in with a CoreDNS instance already to handle pod DNS queries.

Create a separate CoreDNS instance to handle split-horizon zone queries.
Update the existing kube-dns instance (if it’s CoreDNS)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


isClusterService: false
servers:
 - plugins:
 - name: errors
 - configBlock: lameduck 5s
 name: health
 - name: ready
 - name: prometheus
 parameters: 0.0.0.0:9153
 - name: loadbalance
 - configBlock: |-
 stubzones
 path /skydns
 endpoint http://etcd.dns.svc.cluster.local.:2379
 name: etcd
 parameters: home.mydomain.com
 port: 53
 zones:
 - zone: home.mydomain.com
 # Disable TCP unless MixedProtocolLBService=true
 # https://github.com/kubernetes/kubernetes/issues/23880
 scheme: dns://
 use_tcp: false

If the downstream DNS server is located outside the cluster, then enable the LB. If you’re running something like Pi-hole in the cluster, then you don’t need this and can route directly here

1
2
3


service:
 externalTrafficPolicy: Local
serviceType: LoadBalancer

After that, you should have a Load Balancer created like below. Use this to configure your DNS server to forward queries for your internal domain to the LB IP address.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: v1
kind: Service
metadata:
 name: split-horizon-dns-coredns
 namespace: dns
spec:
 type: LoadBalancer
status:
 loadBalancer:
 ingress:
 - ip: 192.168.6.2 # <-- Changed

cert-manager

Cert-manager is responsible for requesting certificates from Let’s Encrypt (or any other compliant ACME certificate provider) and storing them in your cluster. To install, following the standard Helm installation process, but make sure to update following values.

The --dns01-recursive-nameservers flag tells cert-manager not to use the internal DNS server to verify propagation of the DNS01 verifier and instead hit Google’s public DNS server. Feel free to use any public resolver.

1
2
3


extraArgs:
 - --dns01-recursive-nameservers=8.8.8.8:53,8.8.4.4:53
installCRDs: true

After that, create a ClusterIssuer or Issuer. In the DNS01 solver, you’ll need to configure a DNS solver for your domain name. See the official docs on how to do that for your case.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: home-issuer
spec:
 acme:
 email: user@example.com
 preferredChain: ""
 privateKeySecretRef:
 name: cert-manager-info
 server: https://acme-v02.api.letsencrypt.org/directory
 solvers:
 - dns01:
 # Point this to your DNS provider
 selector:
 dnsNames:
 # Replace with your domain zone
 - home.mydomain.com
 - '*.home.mydomain.com'

Router Configuration

After the service is deployed, update the router to forward traffic for your given domain name to the newly created service IP address.

In the Ubiquiti EdgeRouter config, this is defined as:

set service dns forwarding options server=/home.mydomain.com/192.168.6.2

dnsmasq:

server=/home.mydomain.com/192.168.6.2

Future Work

After this is setup, you should be able to access any ingress you’ve got configured under your home DNS zone when you’re inside your home network.

If you want to be able to access your home lab services outside your home network, you can use Wireguard or expose NGINX to the internet and create DNS records in a publicly accessible DNS zone. If you expose any services to the internet, do take care to ensure that you securely restrict access. In a future post, I may document how I do this for my own network.

Best Practices for Java testing with JUnit

Tue, 14 Jun 2022 00:00:00 +0000

JUnit is a popular testing library for Java applications and I extensively used it when working at Amazon for the numerous Java applications and services there. However, I came across a number of different anti-patterns and areas to improve the quality of the test code. This post introduces many of the different tricks and patterns that I’ve learned and shared with my coworkers, and now want to share

Another library to know and reference is Mockito, which I use extensively in JUnit test cases and will reference this too below.

These are all real things that I’ve seen developers do.

Migrate from JUnit4 to JUnit5

If you’re still using JUnit4, why should migrate? Much of this guide will reference features in JUnit5.

JUnit5 has been out since 2017
JUnit5 removes several testing paradigms that contributed to broken test cases
JUnit5 test suites can be be run in the same package with JUnit4 test cases allowing you to slowly migrate (reference)
JUnit5’s Extension API works much better than the runner and can be used to compose different cross-cutting testing concerns

https://junit.org/junit5/docs/current/user-guide/#migrating-from-junit4

Good test cases give meaningful error messages

A failing test case that doesn’t tell you why it failed is not very developer friendly.

Test cases are often times capturing business logic and decisions about how code should behave. For example, they check that action X happens when Y case, but sometimes in the future it’s hard to understand why X should happen in that case. When it’s not obvious why the test case is asserting a situation, provide a useful Javadoc above the method or above the assertion to clarify to other developers.

Remember: Just because you know why it exists today, doesn’t mean you’ll remember why you did something 6 months down the road.

1
2
3
4
5
6
7
8


/**
 * CorporateId is a mandatory field on the Device entity. Without it
 * we'd fail the Frobinator process, thus the DAO must reject it.
 **/
@Test
void testNoRecordSavedWhenCorporationMissing() {
 // ...
}

Prefer using assertEquals and friends over assertTrue

Given the following two test cases that both assert on the size of an array:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


@Test
void testAssertTrue() {
 List<Long> myList = new ArrayList<>();
 Assertions.assertTrue(myList.size() == 1);
}


@Test
void testAssertEquals() {
 List<Long> myList = new ArrayList<>();
 Assertions.assertEquals(1, myList.size());
}

Which error message is easier to read?

1
2
3
4
5


org.opentest4j.AssertionFailedError:
Expected :true
Actual :false

 at FooTest.testAssertTrue(FooTest.java:7)

Or this one using assertEquals?

1
2
3
4
5


org.opentest4j.AssertionFailedError:
Expected :1
Actual :0

 at FooTest.testAssertEquals(FooTest.java:13)

When assertTrue fails, it doesn’t tell you why it failed. You have to go to the line of code to understand why it failed. Imagine if you had five test cases that all failed and they all said gave no useful message. It would take a long time to fix the problem.

Instead, take a look at the Assertions class in JUnit5 and find a relevant method that best matches your assertion. Some examples:

assertArrayEquals
assertEquals
assertInstanceOf
assertNotEquals
assertNotNull
assertSame
assertThrows

If assertTrue is your only option, provide an assertion message (see next item) to help clarify the problem.

Provide assertion messages when the problem is non obvious

An assertion failure message becomes non-obvious when the message does not clearly convey what property is being compared.

1
2
3
4
5


@Test
void testAssertEquals() {
 List<Long> myList = new ArrayList<>();
 Assertions.assertEquals(1, myList.size());
}

When this fails, it just states the expected value is 1, but actual is 0. It doesn’t say why.

1
2
3
4
5
6
7


@Test
void testAssertEquals() {
 List<Long> myList = new ArrayList<>();
 myList.add(5L);
 Assertions.assertEquals(1, myList.size(),
 "The array is supposed to contain one item because we added one to it.");
}

Adding a message at the end can help clarify the problem to the developer. It’s not required to add messages to all assertions.

Use assertAll when testing different properties on an entity

The assertAll method is a special assertion that enables you to perform multiple asserts and fail if any of them failed. If multiple assertions fail, then it’ll print out all failed assertions making it easy to see problems at a glance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


void testAssertEquals() {
 MyObject object = new MyObject("test", "foobar");

 Assertions.assertNotNull(object);
 // Further asserts depend on the above, so it must be separated out
 Assertions.assertAll(
 () -> Assertions.assertEquals("testf", object.first, "First field"),
 () -> Assertions.assertEquals("test", object.second, "Second field")
 );
}

When this fails, it clearly states all the problems at once so I can tackle them instead of fixing one thing, rerunning the tests, then fixing the next problem, until it finally goes green:

1
2
3


org.opentest4j.MultipleFailuresError: MyObject check (2 failures)
 org.opentest4j.AssertionFailedError: First field ==> expected: <testf> but was: <test>
 org.opentest4j.AssertionFailedError: Second field ==> expected: <test> but was: <foobar>

Note that you shouldn’t put all assertions into a single assertAll method. If any assertions depend on previous results, for example I need to separate out into multiple phases of assertions. Otherwise, the future assertion failures provide more and more meaningless messages.

1
2
3
4
5
6
7


Assertions.assertNotNull(object);
Foo foo = object.getFoo();
Assertions.assertNotNull(foo);
Assertions.assertAll(
 () -> Assertions.assertEquals("baz", foo.getBaz()),
 () -> Assertions.assertEquals("bar", foo.getBar())
);

Don’t verify inside a finally block

In Java, finally blocks are executed even if an exception is thrown. If your block of code that you’re testing fails an assertion or throws an exception, then running more assertions in the finally block will mask the original exception and instead will show you verification failure exception. This will mask the exception that matters with an exception message that is obviously going to fail because the Code under Test failed.

Bad:

1
2
3
4
5
6
7
8


@Test
public void testSomething() {
 try {
    myObject.callSomething(); // <-- What if this throws?
  } finally {
    Mockito.verify(someObject).didSomething();
  }
}

Instead, avoid running verifications in a finally block and run them after you run your code. This ensures that when your test case fails, you’ll always see the most relevant and useful exception message.

Better:

1
2
3
4
5


@Test
public void testSomething() {
  myObject.callSomething();
  Mockito.verify(someObject).didSomething();
}

Test Case Accuracy

A test case must be able to fail

Some developers will just write a unit test that covers their newly written code to get the code coverage, then think that’s a sufficient test. A test case that doesn’t fail isn’t useful and even worse, if it doesn’t properly catch bugs, then it gives a false sense of security that the business logic does work correctly.

Ensure that your test cases do fail when your code has bugs or problems. Try introducing an issue and seeing if your test cases fail. Another strategy is TDD (Test Driven Development.) In this paradigm, you write test cases first that refer to code that doesn’t work and implement assertions, then write the code to make the test cases pass.

The PIT Mutation Testing framework is another strategy to ensure that your test cases are effectively testing code. When you run a PIT test against your unit tests, it’ll modify the production code randomly and verify that a test case fails.

IntelliJ Inspection Name: Java -> JUnit -> JUnit test method without any assertions

Don’t use @Test(expected = *Exception.class) (JUnit4)

In JUnit4, it’s common to write unit tests that look like this to test that your code throws exceptions in error cases:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


@Test(expected = NullPointerException.class)
public void testArgumentAssert() {
 // If a NullPointerException is thrown ANYWHERE in this test, it'll pass.

 // Initialization works
 something.testFunction(); // What if "something" was null?

 object.callSomething(null);

 // If the exception is thrown, this never executes
 Mockito.verify(something).importantMethod();
}

However, this introduces the risk of false test passes, i.e. the test can pass when it should fail.

NullPointerException is commonly thrown with parameter validators (e.g. Lombok’s @Nonnull) if the caller passes in a null for a parameter, but it’s also thrown if you call a method on a null method. Devs often times want to validate these parameter validators, but since NPE can mean a variety of things, their test cases end up being low value.

I also frequently see developers expect a RuntimeException, but this has many subclasses. If you expect this type, how do you know it’s what you expected?

Additionally, there’s a Mockito verification after the exception is thrown. This line will never execute, so your Mockito verification is entirely worthless.

Better:

Instead, upgrade to JUnit 5 and use the new Assertions.assertThrows method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


@Test
public void testArgumentAssert() {
 // Initialization works
 something.testFunction();

 // We guarantee that only NPEs thrown on this line are considered passes
 NullPointerException ex = Assertions.assertThrows(NullPointerException.class, () -> object.callSomething(null));

 // With the exception, we can verify that it's the exception that we expected.
 Assertions.assertEquals("Argument foo should not be null", ex.getMessage());

 Mockito.verify(something).importantMethod();
}

With a handle to the actual exception, you can know that it was thrown on the line you expected, however some care needs to be made still that you’re catching what you expect.

Reducing Boilerplate

Create helper test methods

Test case readability matters. Sometimes a test suite will contain a lot of test case methods that all create test harnesses, create mocks, walk through test flows, or perform validations and they end up looking the same over and over again.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


@Test
public void testSomeCase1() {
 Mockito.doReturn(xyz).when(fooBar).someCall(foo);
 Mockito.doReturn(abc).when(xyz).someCall();
 // more test case initialization

 AResult myResult = myClass.performImportantAction(fooBar, 1);
 // Some test code

 Assertions.assertEquals("expected", myRequest.getField());
 // more Assertions
}

@Test
public void testSomeCase2() {
 Mockito.doReturn(xyz).when(fooBar).someCall(foo);
 Mockito.doReturn(abc).when(xyz).someCall();
 // more test case initialization

 AResult myResult = myClass.performImportantAction(fooBar, 2);
 // Some test code

 Assertions.assertEquals("expected", myRequest.getField());
 // more Assertions
}

@Test
public void testSomeCase3() {
 Mockito.doReturn(xyz).when(fooBar).someCall(foo);
 Mockito.doReturn(abc).when(xyz).someCall();
 // more test case initialization

 AResult myResult = myClass.performImportantAction(fooBar, 3);
 // Some test code

 Assertions.assertEquals("expected", myRequest.getField());
 // more Assertions
}

Sure, we were able to add test coverage that verified the code worked, but it’s an unreadable mess. Code reviewers won’t be able to read it to ensure it’s doing the right thing, other developers won’t be able to understand it. Instead create reusable methods.

In the below example, I moved all common logic out to separate methods. Mocks that are needed for all test cases go into the beforeEach, mocks needed only for some methods go into a private method that is then called depending on the test case, then wrapper method is created to call the target method and perform common assertions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


@BeforeEach
void beforeEach() {
 // Place common initialize code here.
 // JUnit calls it before every test case
 Mockito.doReturn(xyz).when(fooBar).someCall(foo);
 Mockito.doReturn(abc).when(xyz).someCall();
}

// If some test cases have differing situations
// Create methods that initial mocked objects
private void mockSpecialCase1() {
 Mockito.doReturn(123).when(xyz).calculateValue();
}

private ARest performActionAndVerify(int case) {
 AResult myResult = myClass.performImportantAction(fooBar, case);

 // Perform any common assertions that always exist for all test cases
 Assertions.assertEquals("expected", myRequest.getField());

 return myResult;
}

@Test
public void testSomeCase1() {
 mockSpecialCase1();

 performActionAndVerify(1);
}

@Test
public void testSomeCase2() {
 AResult myResult = performActionAndVerify(2);
 // Some test code

 Assertions.assertEquals("something", myRequest.getAnotherField());
 // more Assertions
}

@Test
public void testSomeCase3() {
 AResult myResult = performActionAndVerify(3);

 Assertions.assertEquals(5, myRequest.getAnotherField());
}

Each test case becomes easier to read as there’s less irrelevant code in each method.

JUnit5 Extensions

Do you find yourself writing unit test classes that contain lots of the same initialization or tear down logic? In the previous examples, we discussed options for duplicating within a single class, but sometimes multiple classes all have to do some work that isn’t the goal of the test class.

For example, when unit testing a service that emits metrics or X-Ray traces, you may end up with a bunch of code responsible for initializing, collecting, and verifying that those metrics are emitted from many different classes. Each test class itself shouldn’t have to handle this logic and instead should delegate to a common class.

Instead of multiple classes all looking like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


public class FooControllerTest {
 private FooController fooController = new FooController();

 @BeforeEach
 void beforeEach() {
 // Initialize metrics
 // Create mocks
 }

 @AfterEach
 void afterEach() {
 // Tear down metrics
 }
}

public class BarControllerTest {
 private BarController BarController = new BarController();

 @BeforeEach
 void beforeEach() {
 // Initialize metrics
 // Create mocks
 }

 @AfterEach
 void afterEach() {
 // Tear down metrics
 }
}

For this, JUnit provides the extension API that provides many different places to hook into the test runner. Here’s how an example extension can cleanup a class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public class FooControllerTest {
 private FooController fooController = new FooController();

 // MetricsExtension is a custom extension that you write
 // that handles mocking 
 @RegisterExtension
 static MetricsExtension metricsExtension = new MetricsExtension();

 @Test
 void testApiMethod() {
 fooController.callApiMethod();

 // --- Assertions
 // Extension classes can be used to handle assertions too
 metricsExtension.assertMetricsEmitted("DatabaseSuccess", 1);
 }
}

Appendix

Enabling IntelliJ Inspections

IntelliJ’s inspections provide a number of extra static analysis checks that you can enable to catch bugs. To enable one specified in this blog post, go to File -> Settings -> Editor -> Inspections, then find the mentioned inspection.

All testing related inspections I have enabled:

Java -> JUnit
- assertEquals() called on array
- JUnit test method in product source
- JUnit test method without any assertions
- JUnit 5 malformed @Nested class
- JUnit 5 malformed repeated test
- Malformed setUp() or tearDown()
- Malformed @Before or @After method
- Malformed @BeforeClass@BeforeAll
- Malformed test method
- Parameterized test class without data provider method
- Test class with no tests

How to build a useful service data change audit log

Wed, 25 May 2022 00:00:00 +0000

If you’ve got a service that provides clients with the ability to make changes to those entities, then you probably want an audit log that tracks who makes what changes.

I decided to write this post because I frequently saw teams at Amazon not thinking through these considerations. Some of the guidance does focus on AWS IAM, but a lot of it is practical for any type of audit log.

Important aspects to an audit log:

Who made the change?
When did they make the change?
Where did they make the change?
What did they do?

Not all audit logs are the same. I predominately worked in services that owned some kind of data, then needed to track who and what changes were made to its data in a SOA (Service Oriented Architecture) environment. An audit log that tracks user sign-ins or other types of operations may not have any changes to an entity, but they still have a Who, When, and Where.

Terminology

Principal - An authenticated entity that can make calls to your service. Can be a person, computer, or a service
Service Principal - A Principal that represents a service. Since a service may be made up of more than one host/process/AWS Account, this uniquely represents the service
User Principal - A Principal representing a human.

Characteristics of the Log Itself

Append Only

Audit Logs are built for security purposes to know who made a change. If a malicious actor can rewrite the audit log to hide their tracks. An audit log must be append only. It should not be possible for any entries to be deleted or modified. The service should only have the ability to write new entries.

Consider the Domain instead of Storage

I wrote an earlier post about the difference between domain models and storage models that is relevant. The domain model are the core classes that you implement business logic upon, whereas the storage model is what that domain model when forced into the limitations that a given data store requires.

Your audit log should be composed of changes to the domain, not based on the storage model.

Versioned

Take care when designing the schema for your audit log, since it has a longer lifetime than your storage schema. With a database, you can run migrations to upgrade the schema, but for security audit logs may be append only and not allow any writes.

Who made the change?

Who called you is really what person or system made the change. This identity should come directly out of your authentication system It may seem easy at first, but make sure you don’t make any bad assumptions.

Find a meaningful service identity

When you’re processing a request, you need to think about your authentication system to figure out what types of identities you have. Does your authentication handler give you a user id or is it generic like an AWS IAM ARN?

If you use AWS IAM for all calls to your service both human and services with something like Cognito (which I don’t recommend using,) then you’ll get several different pieces of information such as an ARN (eg. arn:aws:iam::1234567890123:root) and session tags. Use this information to convert to something meaningful like a user id or map the account id to a known service name.

arn:aws:iam::1234567890123:assume-role/UserAccessRole_ + STS Session Tags -> userid:2345 (user: foobar123)

Don’t assume that all callers are the same type (e.g. always human)

I’ve frequently seen services just store a username as a string in the event log:

1
2
3
4
5


class AuditLogEntry {
 // ...

 String username // Example: 'user@example.com'
}

What happens if your service also allows non-humans, such as service callers? If you used to store a username like “foobar123” in username, now trying to store a service identity like “InventoryForecastingService/Prod” in the same field is not useful because you don’t know if it’s a human or service.

Instead, distinguish between identities from different identity providers by including a type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


// Provide a different type for all unique identity providers. 
enum ActorType {
 USER,
 SERVICE
}

class ActorIdentity {
 ActorType type
 String identity
}

class AuditLogEntry {
 ActorIdentity actor
}

Consider systems acting on behalf of another identity

Sometimes services will need to call another service, but will be acting on behalf of another identity, such as a user. This might happen in a transitive service call, when a user calls a service that then needs to call another service:

When Service B receives the request, it’s important that it logs both identities since Service A calling it is different than the user calling directly.

There’s two different ways of representing this:

The primary caller is ServiceA and an optional OnBehalfOf attribute denotes the transitive user
The primary caller is UserA and an optional Via attribute denotes the service that made the call

I’ve built systems with both cases, but I prefer the second solution because generally authorization systems primarily evaluate the user permissions and service permissions are coarse grained and the relevant principal is the user.

Make sure to log both principals to the log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// Provide a different type for all unique identity providers. 
enum ActorType {
 USER,
 SERVICE
}

class ActorIdentity {
 ActorType type
 String identity
}

class AuditLogEntry {
 ActorIdentity actor
 Optional<ActorIdentity> via;
}

Example from AWS CloudTrail

For a practical example, we can take a look at how AWS CloudTrail exposes their identities to AWS customers (docs).

Note how depending on the type, there are different relevant attributes:

1
2
3
4
5
6
7


"userIdentity": {
 "type": "Root",
 "principalId": "1234567890123",
 "arn": "arn:aws:iam::1234567890123:root",
 "accountId": "1234567890123",
 "accessKeyId": "ASIABCDEFGHIJKLMNOPQ"
}

When an internal AWS service calls your service, it doesn’t provide any of the internal details and instead exposes a service principal. While internally they’re using IAM, this is an example of how the low-level IAM identity may get mapped into a more meaningful identity when stored in a log:

1
2
3
4


"userIdentity": {
 "type": "AWSService",
 "invokedBy": "compute-optimizer.amazonaws.com"
}

Privacy laws like GDPR will come into play when you’re storing a user’s actions in a log. Often times usernames are the user’s email address which has two problems: it can change and they are considered personal information.

Instead of storing a username or email address in the log, use a unique, non-changing user id that refers to the actual user in another identity service. This reduces the number of services that store sensitive personal information.

Where did they make the change?

Now we know who supposedly made the change, but what happens if a malicious actor secured valid credentials to a user? The next part is including relevant information to aid any investigation.

Some information to include:

Timestamp
IP Address
User Agent
Session Id

What change did they make?

The next problem to solve is to figure out how to represent what mutations they made on the models.

Audit logs should be thought in terms of the domain, not the storage model. The domain model are the core classes that you implement business logic upon, whereas the storage model is what that domain model when forced into the limitations that a given data store requires. I wrote an earlier post about the difference between domain models and storage models that is relevant.

Snapshot

A snapshot based log stores a snapshot of the entity at each version.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


v1:
{
 "GivenName": "John",
 "FamilyName": "Jones",
 "Email": "jpjones2@technowizardry.net",
 "Entitlements": [ "CAN_READ", "CAN_WRITE", "CAN_DELETE" ]
}

v2:
{
 "GivenName": "John",
 "FamilyName": "Jones",
 "Email": "jpjones@technowizardry.net",
 "Entitlements": [ "CAN_READ", "CAN_WRITE", "CAN_DELETE" ]
}

v3:
{
 "GivenName": "John Paul",
 "FamilyName": "Jones",
 "Email": "jpjones@technowizardry.net",
 "Entitlements": [ "CAN_READ", "CAN_WRITE", "CAN_FOOBAR" ]
}

Snapshots make it easy to revert and back to prior revisions, but to actually figure out what changes in between v2 and v3, you need to grab both versions then diff each field. This can get quite complex if you have complex objects with nested fields or arrays.

Diff

A diff based log stores only the values that changed from the previous version.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


v1 (All fields because it's the first version)
{
 "GivenName": "John Paul",
 "FamilyName": "Jones",
 "Email": "jpjones2@technowizardry.net",
 "Entitlements": [ "CAN_READ", "CAN_WRITE", "CAN_DELETE" ]
}

v2:
{
 "Email": "jpjones@technowizardry.net"
}

v3:
{
 "GivenName": "John Paul",
 **"Entitlements": ???**
}

Reading through from v1 to v3, we see that the first entry includes all fields because it’s newly created. The second entry only contains the email change.

But how do we represent the third change? Primitive types are easy to represent, but how do you represent an array? If you represent it as ["CAN_READ", "CAN_WRITE", "CAN_FOO"], the new entire value, then it’s effectively a snapshot at the field level and you need to still do a field level diff to identify the change.

Another way is to break down the changes to arrays into additions and removals:

1
2
3
4
5
6
7
8


v3:
{
 "GivenName": "John Paul",
 "Entitlements": {
 "Added": ["CAN_FOO"],
 "Removed": ["CAN_DELETE"]
 }
}

Now the audit log clearly states what *changed* in v3 of the log. Other complex types can be broken down similarly.

How does this relate to Event Sourcing and CQRS?

In conventional data store patterns, the service will read and write to the same entity in the database with no separation. The database does not inherently maintain a log of changes to a given entity and an audit log must be created independently of the main entities.

Event sourcing is an architectural pattern in which the data store actually maintains the events and changes to a given entity. The current state of an entity is discovered by replaying all of the events from the start. For performance reasons, event sourcing systems often times create periodic snapshots so the entire store doesn’t have to be scanned.

This has the advantage that the change log inherently represents most of the information in an audit log (the change to the entity) and just needs to be supplemented with identity information to become useable.

While this inherent audit log behavior is quite valuable, there are a few disadvantages. In a sample architecture built upon AWS DynamoDB:

Worse Consistency - While data stores like DynamoDB do have eventual consistency where a write may not be instantly seen by a read afterwards, this can become even more prevalent in event sourcing data stores (especially if you build them on-top of other data stores.)

Each event must be strictly serializable such that event B comes after event A. In the above sample architecture, this means that when you create a new event you must get the latest snapshot and apply all pending events before you can create a new event.

Lack of Type Safety through Strong Domain Models - As I mentioned in my blog post about domain model safety, my preferred strategy is to develop strictly validated domain model classes that implement business logic. Only when the entity is saved is it converted to a loose storage model and saved to the data store. This becomes very challenging in the above sample architecture. Any validations must be performed when the event is created and before it’s saved since any invalid customer input should result in a 400 error to the client.

However, the service merely creates events that “request” an entity to be changed. The consumer is the component that actually makes the change and saves it. If the consumer rejects a change then it could block processing of other events and cause a system outage.

Instead, the event should only be created if it results in a valid entity on the output. Unfortunately this can be challenging to implement in code since validation logic needs to be implemented in each command that creates an event. There’s no one single, central place that enforces validations.

Architectural Complexity - In the sample architecture, there are more moving parts involved. All of this adds more things you have to monitor, more code written, and more things to deploy. Developer time isn’t free and systems will break in weird ways at 3am on a weekend.

Conclusion - Both strategies, event sourcing as the audit log and separate audit logs, have advantages and disadvantages. I’ve worked in multiple systems and have had a chance to see both in practice. The goal of this post was to introduce some of the differences and help ask questions so you can build the best solution for your project.

How do you identify changes?

Now that we have some characteristics of a good audit log, how do we actually generate audit events in the source code? This is going to depend heavily on your language and frameworks. In my work, I always used in-house built solutions. Automated solutions didn’t really do what I needed. However, some of the below libraries may be valuable:

Java - Javers
Ruby on Rails - paper_trail gem - This one seems to use the snapshot style to store revisions so you have to compute diffs yourself

Conclusion

In this post, I walked through various aspects of useful audit log for services that control access to a data set and allow users to make changes. This should provide a framework of questions and concerns for you to build your own audit log.

Best Practices for working with Google Guice

Wed, 18 May 2022 00:00:00 +0000

Google Guice is a dependency injection library for Java and I frequently used it on a number of Java services. Compared to Spring, I liked how simple and narrow focused on just dependency injection it was. However, I often times saw developers using it in incorrect or non-ideal patterns that increased boilerplate or were just wrong.

These are all recommendations that I’ve accumulated over several years at working at Amazon watching engineers and sometimes myself improperly leverage Google Guice.

Recommendations

When to reuse modules

If you’ve got multiple different components (e.g. a service API, workers, or other processes) then your Guice modules can be reused and shared to reduce boilerplate code.

Break down your bindings into modules with categories:

Application-specific - These are bindings are for a specific application (e.g. FooServiceModule, FooWorkerModule) and are the top level module that imports everything else
Environment-specific - Contains any bindings that make I/O calls that are shared by may differ between apps, for example AWS clients, secret providers, etc.
Feature-specific - Everything else falls in here. Organize them based on related bindings. Like AuthenticationModule, AuditLogModule, FooFeatureModule.

Prefer Interfaces over concrete classes

The purpose of dependency injection is to minimize coupling between different components. If your class binds to concrete classes, then you’re preventing others from swapping out the implementations based on the environment.

Considerations

If it’s impractical to create separate implementations for a given class, such as pure functional class with no I/O or other external dependencies such as a helper class, then it’s okay to inject the concrete class.

If you need to have multiple copies of the same effective class, then use a @Named instead.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


class SpecificFooClient implements FooClient {}

class FooModule {
  SpecificFooClient providesFooClient(ClassA a) {
 return SpecificFooClient(a);
 }
}

class ConsumingClass {
 @Inject
 public ConsumingClass(SpecificFooClient client) {}
}

In the above example, the Guice binding returns a concrete class type instead of an interface. This forces consuming classes to refer to the concrete class instead of the generic interface. When writing unit tests, now you’re forced to construct the real class instead of swapping out a mocked up implementation.

Avoid @Named to signal implementation when irrelevant

@Named bindings can suffer from the same problem as using concrete classes vs interfaces. Don’t use @Named as a way to signal what specific implementation a class is, instead only add @Named when you need to have two copies. For example, use it to delineate two different AWSCredentialProviders to two different AWS accounts.

Example

I found one service that using @Named to signal the effective type of the class. There was a service client that was exposed by the Guice config that was different depending on the process it was running in. A long-running service got a client that had a time limited cached, whereas a short-term scheduled job process got a client that cached for the entire lifetime of the job.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


class FooModule {
 @Named("CACHING_FOO_CLIENT")
  FooClient providesFooClient(ClassA a) {
 return SpecificFooClient(a);
 }
}

// In another module
class FooModule {
 @Named("NON_CACHED_FOO_CLIENT")
  FooClient providesFooClient(ClassA a) {
 return SpecificFooClient(a);
 }
}

In the above example, there was only ever one type of FooClient that should be available in each process, but depending on the process a different FooClient should be loaded. By including the type in the @Named qualification, it caused the rest of the Guice graph to become more complicated because they all had to be aware of the type instead of being agnostic.

In the below alternative, the Guice binding becomes generic to only expose an abstract FooClient and it internalizes the logic to decide the caching strategy. The rest of the code becomes simpler.

Better:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


// In another module
class FooModule {
  FooClient providesFooClient(ClassA a) {
 if (isShortRunningJob()) {
 return LongCachedFooClient(a);
 } else {
 return TimeLimitedCacheFooClient(a);
 }
 }
}

Or create two separate Guice modules for each environment that return the correct type and avoid any runtime configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


class ShortRunningJobModule {
  FooClient providesFooClient(ClassA a) {
 return LongCachedFooClient(a);
 }
}

class LongRunningServiceModule {
  FooClient providesFooClient(ClassA a) {
 return TimeLimitedCacheFooClient(a);
 }
}

Avoid bind(Foo.class).toInstance(new Foo(…))

In the following example, I’m binding the class type Foo.class to an instance of the Foo class, however this defeats the purpose of Guice since I’m not using it to initialize the class. Any dependency that the MessageProcessor has must be initialized outside of Guice

1
2
3
4
5


public void configure() {
  bind(MessageProcessor.class).toInstance(
 new MessageProcessor(this.sqsClientBuilder())
 );
}

Instead, add @Inject to the MessageProcessor class and use Guice to fully instantiate the class.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


class MessageProcessor {
 @Inject
 MessageProcessor(SqsClientBuilder builder) {
 }
}

class MyModule extends AbstractModule {
 public void configure() {
 // This space intentionally left blank because Guice automatically 
 // knows how to initialize it when used
 }
}

With the new approach, I can easily add or remove new constructor arguments without having to update several different points in the code.

Use @Singletons carefully

Imagine if you have a dependency graph that looks like this. We have a root class marked as @Singleton. Only one class was marked as a Singleton, but the Jackson ObjectMapper at the bottom is not marked as Singleton. The Jackson ObjectMapper class is notoriously expensive to construct and has frequently caused massive latency issues in services because they don’t cache it.

MyRootClass was marked as @Singleton because the developer knew it had classes that were expensive and they didn’t want to redundantly create classes.

Problem: If another class refers to MyOtherClass or reuses the Guice module or that binding for other use cases, the ObjectMapper isn’t marked as Singleton. Thus we’d accidentally re-instantiate it each time.

This is an example of a developer poorly communicating their desires through code. If a class needs to be a Singleton, then that binding itself should be marked as

Solution: Don’t depend on your root classes to be marked as @Singleton. Instead use @Singleton on any class that needs to be a Singleton. Don’t mark it on every class, just the ones that care. Guice will automatically figure out which classes need to be recreated and which ones will be instantiated. That way you can re-use shared Guice modules across multiple systems.
Example

1
2
3
4
5
6
7
8


import com.fasterxml.jackson.databind.ObjectMapper;
import javax.inject.Singleton;

class SerializationModule extends AbstractModule {
 void configure() {
 bind(ObjectMapper.class).in(Singleton.class);
 }
}

Use the Prod Stage when your service is deployed

Guice may eager or lazily initialize the Singleton depending on what stage (ie. PRODUCTION or DEVELOPMENT) the Injector was created using. See here for details.

Lazily initialized Singletons can very easily cause a high latency for the initial few requests to the service until Guice initializes all instances. Instead, you want to eagerly initialize all instances. Since this generally happens before the service added to a load balancer you can take as much time as needed to initialize singletons. This also has the benefit of the JVM preloading most of your code base.

Example on how to initialize Guice with eagerly initialized singletons:

1
2
3
4


import com.google.inject.Injector;
import com.google.inject.Stage;

Injector injector = Injector.createInjector(Stage.PRODUCTION, new Module());

Prefer bind() over @Provides or Providers

The AbstractModule.bind() method is only one line of code compared to @Provides or a Provider. It’s far more concise and doesn’t require changes if you add or remove parameters in your constructor.

Only define a @Provides when you specifically have complex initialization logic that can’t be handled by Guice. Creating them increases the amount of boilerplate code that your application includes with no improvement in code readability.

Bad:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


class MyModule extends AbstractModule {
  @Provides
  FooBar providesFooBar(ClassA a, ClassB b, ClassC c, ClassD d) {
 return SpecificFooBar(a, b, c, d);
  }
}

class SpecificFooBar {
 SpecificFooBar(ClassA a, ClassB b, ClassC c, ClassD d) {
 // ...
 }
}

Better:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


class MyModule {
  void configure() {
    bind(FooBar.class).to(SpecificFooBar.class);
  }
}

class SpecificFooBar {
 @Inject
 SpecificFooBar(ClassA a, ClassB b, ClassC c, ClassD d) {
 // ...
 }
}

Don’t add @Inject to a @Provides method

Don’t add @Inject to your @Provides methods. This is entirely meaningless. @Inject defines what constructor on a class Guice will use to construct or what fields Guice should inject post instantiating. It does not affect anything when defined on a @Provides or when defined on a class

Bad:

1
2
3
4
5
6


class MyModuleextends AbstractModule {
 @Provides
 @Inject
 public SomeClass provideSomeClass() {
 }
}

Bad:

1
2
3
4


// Bad (This does nothing)
@Inject
class SomeClass {
}

Better:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


class SomeClass {
 // OK
 @Inject
 private String someField;

 // OK
 @Inject
 public SomeClass(Another classFoo) {
 }
}

1
2
3
4
5
6
7
8


class SomeClass {
 private String someField;

  // OK
  @Inject
 public SomeClass(Another classFoo) {
  }
}

Even Simpler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


@AllArgsConstructor(onConstructor=@__(@Inject))
classSomeClass {
  // OK
  @Inject
 private String someField;

  @Inject
 public SomeClass(Another classFoo) {
  }
}

Avoid @Inject on private fields

Avoid setting @Inject on fields unless you’re defining optional fields with default values (e.g. configuration values.)

Fields with @Inject on them…

can’t be initialized in unit tests without using Guice to construct them. Developers may forget about this and try to instantiate it with new Example() and will be surprised by a NullPointerException later.
are initialized after the constructor runs which means you have “two initialization phases”. This is often unexpected by other developers who expect the constructor to have all the values needed

Bad:

1
2
3
4


class Example {
  @Inject
 private final SomeClass someClass;
}

Better:

1
2
3
4
5
6
7
8


class Example {
 private final SomeClass someClass;

 @Inject
 public Example(SomeClass someClass) {
 this.someClass = someClass;
 }
}

Use Lombok’s AllArgsConstructor to reduce boilerplate

Lombok can reduce the amount of boilerplate in your code if you wish to use it. Here’s how you can use Lombok to create your constructor.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


import javax.inject.Named;
import javax.inject.Inject;

@AllArgsConstructor(onConstructor = @__({@Inject}))
class Example {
 private final SomeClass someClass;

  // An example with @Named
  @Named("MyTestString")
 private final String aTestString;
}

And add to your lombok.config. Without this, the @Named annotation on fields won’t be copied to the constructor. This will mean Guice will try to bind a generic value instead of your @Named() value

1

lombok.copyableAnnotations += javax.inject.Named

Unit Tests

Why would you want to write unit tests for your Guice modules and code?

What does it mean to unit test Guice modules and bindings? What should you test and not test? Here are some examples of what not to do.

The following example unit test directly calls the @Provides methods on a Module. Notice how there’s 3 different unit tests for a single @Provides methods, but this has several issues:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


@Test(expected = NullPointerException.class)
public void getServiceClient_withNullRegion_throwsException() {
    serviceClientModule.getClient(null);
}

@Test(expected = NullPointerException.class)
public void getServiceClient_withSystemEnvNotSet_throwsException() {
   // given
   ENVIRONMENT_VARIABLES.set(ServiceClientModule.SERVICE_AUTH_ROLE, null);

   // when
   serviceClientModule.getClient(REGION);
}

@Test
public void getServiceClient_withValidRegion_returnsValidClient() {
   // given

   // when
 final Interceptor interceptor = serviceClientModule.getAuthInterceptor(REGION);

   // then
   assertThat(interceptor, is(notNullValue()));
}

It’s testing passing nulls into a @Provides method, but this is useless because Guice won’t pass nulls to your @Provides method (unless you specifically configure it to do so.) Instead, Guice itself going to throw an exception that says it can’t find the binding for your @Provides method. Thus you’re testing a path that will never happen.
Every @Provides method is tested independently requiring copy and pasted code to implement it. There’s a lot of work involved just to get code coverage.
By independently testing methods, you’re not really testing to see if the entire dependency graph is valid. For example, what if you had a module like this:
@Test(expected = NullPointerException.class) is extremely bad because an NPE can be thrown for reasons other than what you expected.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


class BadModule extends AbstractModule {
  // ...

  @Provides
 public FooBar providesFooBar(BazBoo baz) {
 return new FooBar(baz);
  }

  @Provides
 public Bar providesBar() {
    // ...
  }
}

You could independently test each method, but then it’s still fail at runtime because there’s no binding for BazBoo, it’s actually called Bar. Thus, we’ve written a lot of code that adds brittle code, but minimal value add. Guice has much smarter validations than your own tests. Don’t repeat them.

Instead, the following would be better.

Key Improvements:

We’re testing the root-level classes and Guice automatically calls any @Provides, Providers, or @Inject=annotated constructors for us validating correctness
Using JUnit5’s @ParameterizedTest feature reduces the amount of test duplication we have
We can mock out classes that can’t be tested

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


// Use JUnit5
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.provider.ValueSource;
import org.junit.jupiter.params.ParameterizedTest;

class ModuleTest {
 @ParameterizedTest
 @ValueSource(classes = {
    // Add your root-level classes here. Guice will automatically initialize the entire dependency graph behind the scenes and validate that all dependencies work correctly.
    SomeClient.class,
    AnotherClient.class
  })
  void canInitializeClients(Class someClass) {
    Injector injector = Guice.createInjector(binder -> {
     binder.install(new ModuleUnderTest()); // Install the module(s) you're testing
      // If you need to mock anything out because it can't be unit tested, then do this:
      binder.bind(AWSCredentialsProvider.class).toInstance(Mockito.mock(AWSCredentialsProvider.class));
    });

 // If your ModuleUnderTest provides objects you need to mock out, then you need to override:
   Injector injector = Guice.createInjector(
      Modules.override(new ModuleUnderTest())
      .with(binder -> {
        // If you need to mock anything out because it can't be unit tested, then do this:
        binder.bind(AWSCredentialsProvider.class).toInstance(Mockito.mock(AWSCredentialsProvider.class));
      }));

 Assertions.assertNotNull(injector.getInstance(someClass));
  }
}

Binding only test cases

If the entire purpose of your Guice module is to make a call to an external service and get data for a binding, then mocking it out doesn’t do much.

Instead, you can do a binding only test case that verifies that all binds and providers are available and bound to working methods without actually calling any of the code. While it’s not testing code, it’s still extremely valuable test case because it can ensure that you’re not missing a @Provides or any other type of binding.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.provider.ValueSource;
import org.junit.jupiter.params.ParameterizedTest;
import com.google.inject.Stage;

class ModuleTest {
  @ParameterizedTest
  @ValueSource(classes = {
    // Add your root-level classes here. Guice will automatically initialize the entire dependency graph behind the scenes and validate that all dependencies work correctly.
    SomeClient.class,
    AnotherClient.class
  })
  void canInitializeClients(Class someClass) {
   // Stage.TOOL means that no Providers actually run, but it's still sufficient to validate
   Injector injector = Guice.createInjector(Stage.TOOL, new ModuleUnderTest());

    Assertions.assertNotNull(injector.getBinding(someClass));
  }
}

Conclusion

In this post, I provide several Guice anti-patterns that I’ve seen in practice across different teams and alternative approaches to avoid these anti-patterns.

For more information, see the Guice wiki page.

Domain names actually end with a period and why that might subtly break your system

Wed, 11 May 2022 00:00:00 +0000

DNS is the protocol that converts domain names like “technowizardry.net” into the IP address of the server that will respond like “144.217.181.222”. In DNS, domain names actually are supposed to end with a period. For example, the URL of this website is not “www.technowizardry.net”, but it’s actually “www.technowizardry.net.” Notice the period at the end.

Where does this come from? If you look at a DNS packet in a packet capture, you’ll see that each query looks something like this:

The queried domain starts right where I’ve highlighted in the above picture. Domain names are separated by each period. In this example, I have 3 separate domain parts: [“www”, “technowizardry”, “net”]. The byte sequence looks like:

1
2


0000 03 77 77 77 0e 74 65 63 68 6e 6f 77 69 7a 61 72 ·www·tec hnowizar
0010 64 72 79 03 6e 65 74 00 dry·net·

The first byte (0x03) means the first label is 3 bytes. Then 3 bytes come ASCII encoded as “www”. The next byte (0x0e) comes meaning 14 bytes for “technowizardry”. Then another 0x03 saying 3 bytes for “net”. Then finally a 0x00 meaning no more labels in the domain name. The array is null terminated. Once you hit a 0 length label, then you’re done. However, this 0x00 also represents a “.” suffix.

Why does this matter?

You might think, I’ve never had to type a dot at the end of a domain name, why should I care? To find out, first thing is to understand how a DNS query gets performed. When you type in a domain name into your browser, it doesn’t directly turn what you typed into a DNS query and see what it returns. This process is more complicated with modern browsers that combine the search bar with the address bar and have to figure out if you mean to search or go to a domain name. I’ll ignore the search part for now.

On Linux, the browser or application will call a method, getaddrinfo, which itself is responsible for performing DNS lookup. This method will then consult /etc/resolv.conf to understand how to perform the query. Something like this:

1
2
3


nameserver 192.168.2.1
search us-west-2.technowizardry.net technowizardry.net
options ndots:1

In the above file, there’s 3 configuration options:

nameserver - Which just states where to forward your queries. This often times come from DHCP
search - This is the DNS search list. Multiple can be specified. This states that queries for “example” may be translated into “example.us-west-2.technowizardry.net” or “example.technowizardry.net” when creating the query
options ndots - If the query contains 1 or more dots in the query, then it’s considered to be fully qualified. If it contains 0, then it’s uses the search list

Now, maybe you’re starting to see the risk. ndots controls whether or not the query is *assumed* to be fully qualified, i.e. complete. Most web users will type in a full domain name and domains will always have at least one dot in them. facebook.com, google.com, etc. All have a dot, so users never notice an issue.

However, there are a few situations where this will break:

The Curse of Email Addresses

At my job, a coworker recently had to figure out how to validate an email address to provide early feedback to a user that it may be wrong. That coworker found the Apache Commons EmailValidator.java class (note that and the validation looked something like:

1
2
3
4
5
6
7
8
9


public boolean isValid(String email) {
 // ...
 if (email == null) {
 return false;
 }

 if (email.endsWith(".")) {
 return false;
 }

Now this is interesting. If you give an email that looks like “user@foo.”, it’s considered to be invalid, however this check is wrong.

It’s technically valid to have an MX record (the DNS record type that denotes where to deliver email) on a Top Level Domain (TLD). Several TLDs actually have it (source):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


.AI => mail.offshore.AI.
.AS => dca.relay.gdns.net.
.BJ => mail6.domain-mail.com.
.CF => mail.intnet.CF.
.DJ => smtp.intnet.DJ.
 => relais2.intnet.DJ.
.DM => mail.nic.DM.
.GP => ns1.nic.GP.
 => ns34259.ovh.net.
 => manta.outremer.com.
.HR => alpha.carnet.HR.
.IO => mailer2.IO.
.KH => ns1.dns.net.KH.
.KM => mail1.comorestelecom.KM.
.MH => imap.pwke.twtelecom.net.
.MQ => mx1-mq.mediaserv.net.
.NE => bow.rain.fr.
 => bow.intnet.NE.
.PA => ns.PA.
.TD => mail.intnet.TD.
.TT => 66-27-54-142.san.rr.com.
 => 66-27-54-138.san.rr.com.
.UA => mr.kolo.net.
.VA => proxy2.urbe.it.
 => john.vatican.VA.
 => paul.vatican.VA.
 => lists.vatican.VA.
.WS => mail.worldsite.WS.
.TD => mail.intnet.TD
.YE => mail.yemen.net.YE.

However, if I try to send an email to user@ai, what does my mail server do? It will call getaddrinfo to look up the MX record for “ai”. Consulting the previous resolv.conf, since there’s no dots, it’ll try to query “ai.us-west-2.technowizardry.net” or “ai.technowizardry.net”– not what I expected.

This kind of configuration is very common in corporate networks. They’ll specify a DNS search path of their corporate domain name so employees can type “www”. Users end up not being able to email this perfectly valid email account because their mail servers end up being misconfigured.

The generally accepted practice in this case (mentioned in the Stack Overflow answer) is to instead email “user@ai.”. Unfortunately this email validator considers this to be invalid.

Technically it should be legal to send email to both “user@example.com” and “user@example.com.” However due to this subtle behavior we end up with non-compliant software configuration.

Now, in 99.99999% of cases, this is probably a mistake by the user, but I’ve had issues when I was running my Postfix in Kubernetes (see next section) and it caused issues because it tried to resolve internal DNS records until I fixed it.

The Curse of Kubernetes

In the previous example, we talked about an issue when ndots is set 1, but what if we dial it up to 5? That’s exactly the case in Kubernetes.

By default, pods running in Kubernetes get a custom /etc/resolv.conf that looks like this:

nameserver 10.43.0.10 # kube-dns instance in the cluster search mail.svc.cluster.local svc.cluster.local cluster.local options ndots:5

This is done because Kubernetes wants you to be able to query for other pods in the same namespace using just the name like “dovecot” -> “dovecot.mail.svc.cluster.local.” or “mysql.datastore” -> “mysql.datastore.svc.cluster.local”.

But what happens if you try to query for “www.google.com”? There’s two dots there, you and I know it’s fully qualified, but unfortunately getaddrinfo doesn’t. Instead it issues queries for:

Everything but the last query all result in NXDOMAINs (DNS response code for no record found.) This happens for every query you make in a Kubernetes container, except if you explicitly query for “www.google.com.” with a trailing dot.

All of this results in a significant amount of DNS traffic flying around my cluster, and we can clearly see the amount of NXDOMAIN responses for cluster.local domains in the following graph. This causes increased latencies for service operations inside the cluster:

sum(increase(coredns_dns_responses_total{rcode=“NXDOMAIN”}[1h])) by (rcode, zone)

Unfortunately, the search list is part of Kubernetes’ service discovery process and can’t be changed across the cluster.

Fixing Kubernetes

The only way to avoid the increased DNS traffic in Kubernetes is to change the ndots configuration for each Pod:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


apiVersion: v1
kind: Pod
metadata:
 namespace: default
 name: dns-example
spec:
 containers:
 - name: test
 image: nginx
 dnsConfig:
 options:
 - name: ndots
 value: "1"

Once I deployed this across a few key deployments, I saw a dramatic decrease in the number of DNS queries that my cluster sent.

Before making this change, check your application to see if it is making any non fully qualified DNS queries, since those could break. Setting ndots to 1 will ensure that same namespace queries like querying A for redis would find redis.samenamespace.svc.cluster.local. That is the only sane way of using DNS queries. If you need to do cross namespace queries, just go fully qualified and do redis.othernamespace.svc.cluster.local.

The Curse of DNS Servers

Trailing dots are important when creating records in a DNS zone too. In BIND and many DNS configuration UIs, when you create a CNAME, it looks something like this:

1
2


google.technowizardry.net. 900 IN CNAME google.com.
^ Domain Name ^TTL ^Type ^ Target

A CNAME is a type of DNS alias, when I query google.technowizardry.net, it should tell the client to go lookup google.com:

1
2
3
4


$ dig google.technowizardry.net

;; ANSWER SECTION:
google.technowizardry.net. 0 IN CNAME google.com

However, in some servers if I leave off the trailing dot in the CNAME, I actually end up seeing:

1
2
3
4


$ dig google.technowizardry.net

;; ANSWER SECTION:
google.technowizardry.net. 0 IN CNAME google.com.technowizardry.net

Thus, trailing dot is again critical.

Conclusion

In this post, I talked about the subtle assumption that everybody makes that domain names don’t need a trailing dot. While most users don’t have to type trailing dots normally because the DNS query library usually fixes the problem for you, it’s technically required and will cause some strange behavior if you’re not aware of it.

References

https://github.com/kubernetes/kubernetes/issues/45976

https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html

Accurate, Local Home Energy Monitoring: Part 2 - Network Config

Sun, 01 May 2022 00:00:00 +0000

This post continues from the previous post in the series where I walked through the decision process on what energy monitor system to use and how to install Brultech GEM Monitor. I ended with the hardware physically installed and all Current Transformers (CTs) connected.

In this post, I continue from that point and walk through the network and software configuration defining each circuit size.

Network Configuration

First, connect the device to the network (I’m using Ethernet) and ensure it’s turned on. Then discover the IP address of the monitor. I found it by logging in to the router’s config web page and checking for the DHCP lease from the device. Brultech has a few different tools that help you discover the device, but they didn’t really work for me and looked like they were written a decade ago.

To configure how to control the device, I port scanned it and found it listens on several different ports:

1
2
3
4
5


PORT STATE SERVICE VERSION
23/tcp open telnet?
80/tcp open http
1025/tcp open NFS-or-IIS?
8000/tcp open http-alt GEM ver1

Port 80 and port 8000 are both HTTP services that control different aspects of the monitor.

The default credentials for port 80 are username:admin and password:admin.

Disabling Wi-Fi

This GEM monitor includes both Wi-Fi and Ethernet connections. By default, it broadcasts a Wi-Fi network (SSID: GreenEye_XYZ) that you can connect to, however if you’re connecting via Ethernet, an extra Wi-Fi network just adds noise to the RF channel. Removing the antenna doesn’t fully stop it. Luckily in COM firmware version v4.17+, you can explicitly disable it (reference.) To disable it, navigate to:

http://{ipaddress}:8000/

and click the “Adv” tab at the top., then scroll down to near the bottom where it has a button for “AP off”

The GEM monitor should reboot and the Wi-Fi network will be gone. The first time I did this, the monitor failed to come back (possibly because I modified Wi-Fi settings on the port 80 before trying to disable it.) If that happens, reset it by holding the network reset button for 5 seconds and going directly to this page and disabling the AP.

Configuring the CTs

Since CTs can come in different sizes (30amps all the way to 200 amps,) the GEM device needs to know exactly what CT is installed on each channel. Otherwise it won’t know how to translate the signals it receives into the correct current measurement.

To assign the channels, download the GEM Network Utility from BrulTech download page. Switch to TCP Client mode, set Port: 8000, and enter the IP address of the device, then click Open on the right side.

If it succeeds, it should show Status: Connected. In the menu bar, click CT and PT Settings. In the window that opens, configure the CT types for each channel then click Save.

After that, check out the Live Data tab to verify everything looks right. Below, we tested turning on several heaters and verified that the values of each channel sum up approximately to the total in channel 1. Before changing the models in the previous steps, the channels were not adding up. After the change, I saw that the individual values didn’t sum up to be exactly equal to the total in channel 1, but it was pretty close and I imagine there’s some measurement tolerances because each channel is being measured independently.

GEM Utility showing lots of energy usage from several different devices confirming that it works.

Next Steps

Now everything should be measured correctly. The next step will be to connect it to HomeAssistant. This will come in the next blog post.

Kubernetes: A hybrid Calico and Layer 2 Bridge+DHCP network using Multus

Wed, 27 Apr 2022 00:00:00 +0000

Previously in my Home Lab series, I described how my home lab Kubernetes clusters runs with a DHCP CNI–all pods get an IP address on the same layer 2 network as the rest of my home and an IP from DHCP. This enabled me to run certain software that needed this like Home Assistant which wanted to be able to do mDNS and send broadcast packets to discover device.

However, not all pods actually needed to be on the same layer 2 network and lead to a few situations where I ran out of IP addresses on the DHCP server and couldn’t connect any new devices until reservations expired:

My DHCP IP pool completely out of addresses to give to clients

I also had a circular dependency where the main VLAN told clients to use a DNS server that was running in Kubernetes. If I had to reboot the cluster, my Kubernetes cluster could get stuck starting because it tried to query a DNS server that wasn’t started yet (For simplicity, I use DHCP for everything instead of static config).

In this post, I explain how I built a new home lab cluster with K3s and used Multus to run both Calico and my custom Bridge+DHCP CNI so that only pods that need layer 2 access get access.

I wanted to move the K8s pods into a separate IP pool and VLAN so I could reduce the blast radius of something going wrong.

If you’re trying to start a K3s cluster (like I am) from scratch, then you may run into issues where the K3s cluster that Rancher provisions won’t start without a working CNI. If this happens, check out my other post on how to install the CNI.

Network Configuration

I created a new VLAN (ID 20) and trunked this VLAN to my router and all switches and configured the router as a DHCP server and enabled it to route traffic between VLANs and the internet.

I tried trunking the VLAN 20 (and left the default VLAN as untagged) to both computers, however I ran in to an issue where the Surface Dock 2 Ethernet adapter wouldn’t work because it couldn’t receive ARP packets from certain devices on the network on the VLAN tagged adapter. This didn’t make any sense because it was able to get an IP address from DHCP.

The router wasn’t able to send ARP responses/queries to the VM, but other machines on the network were able to. Logically the only difference I saw was the packet lengths, packets were always less than 64 bytes, but Ethernet is supposed to pad all bytes to a minimum of 64 bytes. This didn’t make any sense, so instead I bought a USB-Ethernet adapter for this computer and used that for my secondary network.

A packet capture from a tap on the Ethernet cable showing packets, but the responses never made it to the IP stack in the VM

VM Network Adapter Configuration

I’m running my Kubernetes nodes as a Hyper-V VM on two of my Windows computers.

I previously created a virtual switch (See part #1 if you’re interested in the step by step) for Kubernetes that references my Ethernet adapter:

Then I created a virtual machine that includes two network adapters, both of them bound to that same switch, but one of them included the VLAN Id 20

Cluster Configuration

I’m using Rancher’s UI to provision a k3s cluster. By default K3s uses Flannel for it’s networking CNI, but I want Multus so I can use Calico. While creating the cluster, set the agent Env variable, INSTALL_K3S_EXEC to be --flannel-backend=none --disable-network-policy to deploy without the default Flannel cluster, then follow this guide.

1
2
3
4
5
6
7
8
9


apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
 name: example
 namespace: fleet-default
spec:
 agentEnvVars:
 - name: INSTALL_K3S_EXEC
 value: '--flannel-backend=none --disable-network-policy'

OS Network Configuration

Once Linux is booted in the VM, we can setup the network adapters inside the VM. For more details, see my post on Bridge + Systemd.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


$ ls /etc/systemd/network
10-netplan-eth0.network 10-netplan-eth1.network cni0.netdev cni0.network

$ cat /etc/systemd/network/10-netplan-eth0.network
[Match]
Name=eth0

[Network]
Bridge=cni0

$ cat /etc/systemd/network/cni0.netdev
[NetDev]
Name=cni0
Kind=bridge
MACAddress=00:15:5d:02:cb:09 # Same as eth0 MAC address

$ cat /etc/systemd/network/cni0.network
[Match]
Name=cni0

[Network]
DHCP=yes
IPv6AcceptRA=yes

The eth1 adapter should be bound to the VLAN 20 network adapter

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


$ cat /etc/systemd/network/10-netplan-eth1.network
[Match]
Name=eth1

[Link]
MTUBytes=9000 # Optional: Configure Jumbo Frames. All devices on this VLAN need to be configured with this. Since this is limited to just my K8s nodes, this is possible

[Network]
DHCP=yes

# Without this we end up with two different default routes. For security, I want to prefer all traffic to pass over the separate VLAN. This VLAN will bypass certain restrictions that I have configured on my main VLAN like forcing all DNS traffic to go through my pi-hole

[Route]
Destination=0.0.0.0/0
Gateway=192.168.3.1
Metric=90

Now I end up with the following route table:

1
2
3
4
5
6
7
8


$ ip route
default via 192.168.3.1 dev eth1 proto static metric 90 onlink
default via 192.168.3.1 dev eth1 proto dhcp src 192.168.3.2 metric 1024
default via 192.168.2.1 dev cni0 proto dhcp src 192.168.2.151 metric 1024
192.168.2.0/24 dev cni0 proto kernel scope link src 192.168.2.151
192.168.2.1 dev cni0 proto dhcp scope link src 192.168.2.151 metric 1024
192.168.3.0/24 dev eth1 proto kernel scope link src 192.168.3.2
192.168.3.1 dev eth1 proto dhcp scope link src 192.168.3.2 metric 1024

Intro to Multus

Multus is a special CNI that enables you to configure one or more network interfaces on a pod’s network namespace. Each pod always gets the default cluster network/master plugin interface. Then pods can add a special annotation to get more network adapters.

I’m going to use Calico as my cluster network plugin because it supports BGP and is what I was already using based on previous posts in my series.

My DHCP CNI will be the optional secondary network attachment.

Installing Multus

Unfortunately, Multus doesn’t currently provide any Helm templates. Instead they only provide a YAML file that needs to be modified because it can be used.

First, download the multus-daemonset.yml from their GitHub repository and save it.

Find the ConfigMap that defines multus-cni-config. This is what defines the primary network plugin.

Since I’m using Calico, I used the following ConfigMap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63


kind: ConfigMap
apiVersion: v1
metadata:
 name: multus-cni-config
 namespace: kube-system
 labels:
 tier: node
 app: multus
data:
 cni-conf.json: |-
 {
 "name": "multus-cni-network",
 "type": "multus",
 "cniVersion": "0.3.1",
 "capabilities": {
 "portMappings": true
 },
 "delegates": [
 {
 "name": "calico-network",
 "cniVersion": "0.3.1",
 "plugins": [
 {
 "type": "calico",
 "datastore_type": "kubernetes",
 "mtu": 0,
 "nodename_file_optional": false,
 "log_level": "Info",
 "log_file_path": "/var/log/calico/cni/cni.log",
 "ipam": {
 "type": "calico-ipam",
 "assign_ipv4": "true",
 "assign_ipv6": "false"
 },
 "container_settings": {
 "allow_ip_forwarding": false
 },
 "policy": {
 "type": "k8s"
 },
 "kubernetes": {
 "k8s_api_root": "https://10.43.0.1:443",
 "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
 }
 },
 {
 "type": "bandwidth",
 "capabilities": {
 "bandwidth": true
 }
 },
 {
 "type": "portmap",
 "snat": true,
 "capabilities": {
 "portMappings": true
 }
 }
 ]
 }
 ],
 "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
 }

Then find the kube-multus-ds DaemonSet and change both the args and the volumes section to look like below. This forces Multus to run before Calico

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


 containers:
 - name: kube-multus
 image: ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9.3
 command: ["/entrypoint.sh"]
 args:
- - --multus-conf-file=auto"
+ - --multus-conf-file=/tmp/multus-conf/0-multus.conf
 - "--cni-version=0.3.1"
----
 volumes:
 - configMap:
 defaultMode: 420
 items:
 - key: cni-conf.json
- path: 70-multus.conf
+ path: 0-multus.conf
 name: multus-cni-config

Install Calico

Now you can install Calico as you normally would. I’ve already got a guide on how I configure Calico in my network here.

When installing using the Tigera Operator, make sure to configure the nodeAddressAutodetectionV4/V6 settings to use the VLAN 20 (in my case eth1.)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
 name: default
spec:
 calicoNetwork:
 bgp: Enabled
 ipPools:
 - blockSize: 26
 cidr: 192.168.7.0/24
 encapsulation: None
 natOutgoing: Disabled
 linuxDataplane: Iptables
 nodeAddressAutodetectionV4:
 canReach: eth1
 cni:
 ipam:
 type: Calico
 type: Calico
 controlPlaneNodeSelector:
 node-role.kubernetes.io/master: "true"
 nodeUpdateStrategy:
 rollingUpdate:
 maxUnavailable: 1
 type: RollingUpdate
 nonPrivileged: Disabled
 variant: Calico

After installing Calico, the cluster should start up correctly and you should be able to launch pods with at least internet connectivity. Next, we need to configure the layer 2 network CNI.

Install the Layer 2 Bridge CNI

Now install the DHCP CNI and DaemonSet that I’ve been working on in previous posts (see here):

github.com/ajacques/cni-plugins/…/dhcp/k8s.yaml:

kubectl apply -f https://raw.githubusercontent.com/ajacques/cni-plugins/bridge/plugins/ipam/dhcp/k8s.yaml

Then create a Multus NetworkAttachment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
 name: layer2-bridge
 namespace: default
spec:
 config: |
 {
 "cniVersion": "0.4.0",
 "name": "dhcp-cni-network",
 "plugins": [
 {
 "type": "bridge",
 "name": "mybridge",
 "bridge": "cni0",
 "isDefaultGateway": false,
 "uplinkInterface": "eth0",
 "enableIPv6": true,
 "ipam": {
 "type": "dhcp",
 "provide": [
 { "option": "12", "fromArg": "K8S_POD_NAME" }
 ]
 }
 }
 ]
 }

Updating the Deployment

Configuring the deployment to use dual network adapters is easy, add the annotation to the pod annotations not the deployment annotations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: apps/v1
kind: Deployment
metadata:
 name: homeassistant
 namespace: smarthome
spec:
 template:
 metadata:
 annotations:
 k8s.v1.cni.cncf.io/networks: default/layer2-bridge

Solving Network Routing Problems

As I encountered previously in the series (in Part 5) the containers have an entirely separate route table

1
2
3
4


sudo ip netns exec {containerns} ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
192.168.2.151 dev net1 scope link

When HomeAssistant tries to send a Wake On Lan packet to turn on my TV at IP address 192.168.2.xy it needs to send a packet to 192.168.2.255. But now our traffic isn’t making it directly out onto the layer 2 network. It matches the default route and goes through the host which is prevents broadcast packets from being broadcast since it’s considered an layer 3 hop and multicast.

We need to tell the container that it can send traffic destined for the main LAN towards to cni0/eth0/VLAN 1 network adapter.

I tried creating a custom route by using the redhat-nfvpe/cni-route-override plugin:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
 name: layer2-bridge
 namespace: default
spec:
 config: |
 {
 "cniVersion": "0.4.0",
 "name": "dhcp-cni-network",
 "plugins": [
 {
 "type": "bridge",
 "name": "mybridge",
 "bridge": "cni0",
 "isDefaultGateway": false,
 "uplinkInterface": "eth0",
 "enableIPv6": true,
 "ipam": {
 "type": "dhcp",
 "provide": [
 { "option": "12", "fromArg": "K8S_POD_NAME" }
 ]
 },
 {
 "type": "route-override",
 "addroutes": [
 { "dst": "192.168.2.0/24" }
 ]
 }
 }
 ]
 }

This allows the container to send traffic through cni0 onto the correct VLAN, but with the wrong source IP and it sends it as 192.168.7.xy (The Calico K8s Pod subnet.) The container route table looks like this:

1
2
3
4
5


$ sudo ip netns exec {containerns} ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
192.168.2.158 dev net1 scope link
192.168.2.0/24 dev net1 scope link

The route is missing a src 192.168.2.xyz to tell the Linux IP stack to use the right source IP address.

I see the same problem with mDNS traffic that HomeAssistant uses to discover devices on the local network. The following are DEBUG logs showing it’s creating a socket to 239.255.255.250, the multicast IP address for mDNS.

2022-04-17 00:19:51 DEBUG (MainThread) [async_upnp_client.ssdp] Creating socket, source: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '192.168.7.253', ('192.168.7.253', 0)), target: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '239.255.255.250', ('239.255.255.250', 1900))

Unfortunately, the route-override CNI plugin doesn’t allow us to define the source field on a defined route, so we have to define our own CNI plugin.

To figure out how to create the right rule, we need the subnet of the loocal network and the network adapter inside the container. Kubernetes stores all of the outputs from the CNI plugins in /var/lib/cni. If we inspect the output file, we can that this information will get passed into a custom CNI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


// cat /var/lib/cni/multus/results/dhcp-cni-network-018684df7309507adb027bbd2b1cec1c06be49d9f1b1ab51601fdb798ce85b7f-net1 | jq
{
 // ...
 "result": {
 "cniVersion": "0.4.0",
 "dns": {},
 "interfaces": [
 {
 "mac": "00:15:5d:02:cb:09",
 "name": "cni0"
 },
 {
 "mac": "3a:34:17:72:61:28",
 "name": "veth4fd7d3d5"
 },
 {
 "mac": "56:c6:ce:2b:6e:f7",
 "name": "net1",
 "sandbox": "/var/run/netns/cni-090e5f88-9a95-cb98-6cd0-1b8b74ebe32f"
 }
 ],
 "ips": [
 {
 "address": "192.168.2.158/24",
 "gateway": "192.168.2.1",
 "interface": 2,
 "version": "4"
 }
 ],
 "routes": [
 {
 "dst": "0.0.0.0/0",
 "gw": "192.168.2.1"
 },
 {
 "dst": "192.168.2.0/24"
 }
 ]
 }
}

The full code for the CNI is here. A break down:

First, we grab a reference to the network namespace for the container (CNI passes this in directly.) linkName will be the name of the container once we’re inside the container and containerNet is 192.168.2.158/24.

1
2
3
4
5
6
7


// Load Configuration (See GitHub for code)

netns, _ := ns.GetNS(args.Netns)
defer netns.Close()

linkName := prevResult.Interfaces[2].Name
containerNet := prevResult.IPs[0].Address

Then swap inside the container network namespace and get a reference to adapter:

1
2


err = netns.Do(func(_ ns.NetNS) error {
 containerLink, err := netlink.LinkByName(linkName)

Next, we need to convert the IP address 192.168.2.158/24 to 192.168.2.0/24 since Linux prohibits the former to be used as part of a route and then add it as a route.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


 // 192.168.2.0/24 dev net1 scope link src 192.168.2.158
 route := &netlink.Route{
 LinkIndex: containerLink.Attrs().Index,
 Scope: netlink.SCOPE_LINK,
 Src: containerNet.IP,
 Dst: &net.IPNet{
 IP: containerNet.IP.Mask(containerNet.Mask),
 Mask: containerNet.Mask,
 },
 }

 err = netlink.RouteAdd(route)

Then create a similar route for multicast traffic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 _, i, err := net.ParseCIDR("224.0.0.0/4")

 mcastroute := &netlink.Route{
 LinkIndex: containerLink.Attrs().Index,
 Scope: netlink.SCOPE_LINK,
 Src: containerNet.IP,
 Dst: i,
 }

 err = netlink.RouteAdd(mcastroute)

This is all taken care of if you use the Docker Image I wrote and update the network attachment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
 name: layer2-bridge
 namespace: default
spec:
 config: |
 {
 "cniVersion": "0.4.0",
 "name": "dhcp-cni-network",
 "plugins": [
 {
 "type": "bridge",
 "name": "mybridge",
 "bridge": "cni0",
 "isDefaultGateway": false,
 "uplinkInterface": "eth0",
 "enableIPv6": true,
 "ipam": {
 "type": "dhcp",
 "provide": [
 { "option": "12", "fromArg": "K8S_POD_NAME" }
 ]
 },
 {
 "type": "route-fix"
 }
 }
 ]
 }

Now, if we redeploy HomeAssistant it successfully discovers devices on my LAN!

Conclusion

In this post, I pulled together several different techniques I applied in previous posts in this series showing how to use Multus to run both Calico and Bridge+DHCP CNIs at the same time. Calico enables us to isolate traffic to a separate VLAN and avoid consuming all the IP addresses in the LAN and Multus with the bridge CNI ensures that software like HomeAssistant, my Sonos control software, and software that uses mDNS can continue to discover devices like they should.

Techno Wizardry

Trying to use LiteLLM Proxy in my smart home

My Requirements

The landscape

LiteLLM

Tool calls are breaking my Home Assistant

Too much Vibe coding

LiteLLM crashes during Ollama

Slow as molasses

Conclusion

Ads and data brokers are out of control

What are data brokers?

Where does the government come in?

Block Ads

Data Brokers

Conclusion

References

Replatforming RKE1 to Nix-based K8s - Part 1

First Cut

SSL

Pause image

Dude, where’s my CNI?

Why’s my service cidr broken?

Pin Kubernetes

Conclusion

Syncing my Christmas lights to my Sonos

How do I connect a speaker?

Control Protocols

Wireshark Dissector

The Control Script

More infrastructure doesn't fix using the wrong infrastructure

Queue Processing Lambda

Reduce concurrency

More problems with throttling

Taking control of the process life-cycle

Conclusion

Blogging on the Fediverse with ActivityPub

What is ActivityPub?

The WebFinger

Generating the outbox

Generate per-post files

Making the Accept header work

Attempt 1 - Using NGINX

Conclusion

References

I like the idea of Nix, but don't enjoy using it

The Good Parts

The Bad Parts

Nix, the language

Variables

Nix, the CLI

Errors

Confusing Commands

Nix, the package manager

What I would have expected for versioning

How do I even override something?

How do I pin NixPkgs?

Conclusion

Proxy ARP is broken on Unifi U7 Lite

First Attempt using airomon-ng

Decrypting WPA3

A second look at the HostAPd logs

Is it DNS?

tcpdump everything

Proxy ARP

Auto enable user namespaces in Kubernetes

Release Schedule

Thinking about policies

Creating the policy

Mutate all the things

Skip for pods with other host namespacing

Problems with PVCs

Skip for unsupported volumes

Skip host hardware devices

Skip for Longhorn

The Final Policy

See it in action

Guaranteed Quality of Service in my Home Lab

The problem

Resource requests and limits