diff --git a/flake.nix b/flake.nix index 58facc2..633a41f 100644 --- a/flake.nix +++ b/flake.nix @@ -35,8 +35,7 @@ | .key as $parent_key | .value.inputs | keys[] | select(IN($root_input_keys[])) - | {"key": $parent_key, "value": .} - | .key + ".inputs." + .value + ".follows = \"" + .value + "\";"' + | $parent_key + ".inputs." + . + ".follows = \"" + . + "\";"' ''; runtimeInputs = [pkgs.jq]; }) diff --git a/machines/moritz-server/website/root/.gitignore b/machines/moritz-server/website/root/.gitignore index 98478da..7b82297 100644 --- a/machines/moritz-server/website/root/.gitignore +++ b/machines/moritz-server/website/root/.gitignore @@ -1 +1,2 @@ themes/ +public/ diff --git a/machines/moritz-server/website/root/config.toml b/machines/moritz-server/website/root/config.toml index cd3df49..b91baf1 100644 --- a/machines/moritz-server/website/root/config.toml +++ b/machines/moritz-server/website/root/config.toml @@ -28,5 +28,5 @@ header_nav = [ { url = "/", name_en = "/home/"}, # { url = "/about", name_en = "/about/"}, # { url = "/journal", name_en = "/journal/"}, - # { url = "/blog", name_en = "/blog/"} + { url = "/blog", name_en = "/blog/"}, ] diff --git a/machines/moritz-server/website/root/content/blog/2025-05-19-nix-follows.md b/machines/moritz-server/website/root/content/blog/2025-05-19-nix-follows.md new file mode 100644 index 0000000..e75f068 --- /dev/null +++ b/machines/moritz-server/website/root/content/blog/2025-05-19-nix-follows.md @@ -0,0 +1,158 @@ ++++ +title="Generate Nix Flake \"follows\"" +template="custom/blog/page.html" ++++ +Like many [Nix](https://nixos.org) users I am using the "experimental" flakes feature[^1]. +Flakes are a way of defining inputs and outputs: +```nix +{ + inputs = { + nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable"; + # other inputs + }; + outputs = inputs @ {nixpkgs, ...}: { + # imagine some output here + }; +} +``` + +The outputs are used (as the name implies) for some kind of output, such as: packages, NixOS configuration, development shells, ...\ +But as this blog post exclusively revolves around the inputs part you can already forget about outputs again. + +The inputs are a reference to a source, like a git repo, which are used by the flake. +Most often you want to have one input for _nixpkgs_, where most of the package definitions for Nix are found. +Additionally you will likely use some Nix libraries which are also inputs to your flake. +You can also track source code of projects which do not use Nix at all, which allows for the equivalent of "-git" packages in the AUR. +To make flakes reproducible the specific version of each input is recorded in the "flake.lock" file. + +## Inputs all the way down +The example flake above is so simple that you will hardly see something alike in the wild. +As a real world example we will look at the flake I use to define the configurations of all my machines. +In it I use many libraries and NixOS modules[^2] which leads to an inputs section like this: +```nix +{ + inputs = { + nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable"; + flake-parts.url = "github:hercules-ci/flake-parts"; + + clan-core.url = "git+https://git.clan.lol/clan/clan-core"; + clan-core.inputs.nixpkgs.follows = "nixpkgs"; # Needed if your configuration uses nixpkgs unstable. + + flake-utils.url = "github:numtide/flake-utils"; + git-hooks.url = "github:cachix/git-hooks.nix"; + home-manager.url = "github:nix-community/home-manager"; + impermanence.url = "github:nix-community/impermanence"; + jovian.url = "github:Jovian-Experiments/Jovian-NixOS"; + lix-module.url = "https://git.lix.systems/lix-project/nixos-module/archive/2.93.0.tar.gz"; + nixos-mailserver.url = "gitlab:simple-nixos-mailserver/nixos-mailserver"; + nur.url = "github:nix-community/NUR"; + niri.url = "github:sodiboo/niri-flake"; + }; + outputs = _: {}; +} +``` +In reality I use even more inputs, but you get the point. + +And to complicate things further all of these inputs seen above are also flakes. +This means they can define inputs, which could in turn be flakes, which could define inputs, and so on. +As eluded to earlier we need to keep track of each inputs version to ensure reproducibility. + +But with so many inputs which also define inputs we inevitably end up with duplicates. +In some cases this can be exactly what we want to ensure we build the same thing as the person who defined the input we use. +However often we do not want 10 different versions of _nixpkgs_ and in turn 10 versions of bash or other packages. +The evaluation time also increases with the number of distinct inputs. + +## I will follow you +To solve this problem we can use the follows feature. +I already used this in the second example to make the _nixpkgs_ of _clan-core_ follow our _nixpkgs_ defined above. +This causes nix to override the _nixpkgs_ version of _clan-core_ to the same one as the _nixpkgs_ input. + +For a handful of inputs this can be easily written by hand, but at some point becomes tiring. +The flake defined in the previous section uses a staggering 49 inputs[^3]. +I definitely do not want to figure out and write all the follow statements for this by hand! + +## Just One Line +To get an overview for inputs which are redundant I wrote a [jq](https://jqlang.org/) one-liner[^4]: +```sh +nix flake metadata --json | jq -r \ + '.locks.nodes + | map(select(has("inputs")) + | .inputs[]) + | flatten + | map(select(test(".*_[0-9]+")))[]' +``` +Resulting in the following candidates for duplicate inputs: +```text +systems_2 +systems_3 +gitignore_2 +nixpkgs_2 +nixpkgs_3 +flake-utils_2 +nixpkgs_4 +nixpkgs_5 +flake-compat_2 +git-hooks_2 +nixpkgs_6 +nixpkgs-24_11 +flake-parts_3 +nixpkgs_8 +treefmt-nix_2 +flake-parts_2 +nixpkgs_7 +``` +In this list `nixpkgs-24_11` is a false positive, but the rest are duplicates. +To avoid these duplicate inputs I wrote another one-liner. +```sh +nix flake metadata --json | jq -r \ + '(.locks.nodes.root.inputs | keys) as $root_input_keys + | .locks.nodes | to_entries[] + | select(.key | IN($root_input_keys[])) + | select((.value | has("flake") | not) or (.value.flake == true)) + | select(.value | has("inputs")) + | .key as $parent_key + | .value.inputs | keys[] + | select(IN($root_input_keys[])) + | $parent_key + ".inputs." + . + ".follows = \"" + . + "\";"' +``` +It generates a follow statement for every non top-level input with the same name as a top-level input. +Looking at our example we already have a top-level input called _nixpkgs_. +This means the script will create a follow statement for every other input called _nixpkgs_ to follow this top-level _nixpkgs_. + +Running this script produces a list of follow statements: +```text +clan-core.inputs.flake-parts.follows = "flake-parts"; +clan-core.inputs.nixpkgs.follows = "nixpkgs"; +git-hooks.inputs.nixpkgs.follows = "nixpkgs"; +home-manager.inputs.nixpkgs.follows = "nixpkgs"; +jovian.inputs.nixpkgs.follows = "nixpkgs"; +lix-module.inputs.flake-utils.follows = "flake-utils"; +lix-module.inputs.nixpkgs.follows = "nixpkgs"; +niri.inputs.nixpkgs.follows = "nixpkgs"; +nixos-mailserver.inputs.git-hooks.follows = "git-hooks"; +nixos-mailserver.inputs.nixpkgs.follows = "nixpkgs"; +nur.inputs.flake-parts.follows = "flake-parts"; +nur.inputs.nixpkgs.follows = "nixpkgs"; +``` + +By adding these to our flake we reduce the number of inputs to 36. + +We can reduce this further by adding new _top-level_ inputs which are used often by the libraries we use. +To do so we can again identify duplicates, add them as top-level imports and run the script again. + +## Conclusion +We were able to reduce our number of inputs from 49 inputs to 36 (which could be reduced even further). +This should speed up evaluation and avoid some duplicate packages for our flake. + +There exist a similar tool [nix-auto-follow](https://github.com/fzakaria/nix-auto-follow) which rewrites the "flake.lock" file directly to remove duplicates. +However by rewriting the lock file it is harder to edit/change what inputs follows which. +There is an [open issue](https://github.com/fzakaria/nix-auto-follow/issues/19) to make it more configurable. +So maybe this problem could be solved in the future. + +For now tough I will just use my trusty one-liner! + +--- +[^1]: by now I doubt they will ever get stable +[^2]: also libraries I guess +[^3]: run `nix flake metadata | grep -v "follows input" | grep "Last modified" -c` and subtract 1 for the flake itself +[^4]: if you ignore line breaks added for readability diff --git a/machines/moritz-server/website/root/content/blog/_index.md b/machines/moritz-server/website/root/content/blog/_index.md new file mode 100644 index 0000000..9884c59 --- /dev/null +++ b/machines/moritz-server/website/root/content/blog/_index.md @@ -0,0 +1,4 @@ ++++ +title="Blog Posts" +template="custom/blog/section.html" ++++ diff --git a/machines/moritz-server/website/root/templates/custom/blog/page.html b/machines/moritz-server/website/root/templates/custom/blog/page.html new file mode 100644 index 0000000..711eea2 --- /dev/null +++ b/machines/moritz-server/website/root/templates/custom/blog/page.html @@ -0,0 +1,5 @@ +{% extends "blog/page.html" %} +{% block content %} +{{ super() }} + +{% endblock %} diff --git a/machines/moritz-server/website/root/templates/custom/blog/section.html b/machines/moritz-server/website/root/templates/custom/blog/section.html new file mode 100644 index 0000000..11b6bb4 --- /dev/null +++ b/machines/moritz-server/website/root/templates/custom/blog/section.html @@ -0,0 +1,2 @@ +{% extends "blog/section.html" %} +