Rust analyzer seems to be one of the few lsp servers which needs a small timeout for inlay hints to show up correctly. All other servers get their inlay hints activated without delay.
Welcome to my dotfiles, where I configure everything thanks to nix. Feel free to poke around and nick everything you like for your own configs.