So I was able to fix this by jumping up to Nvidia driver version 555 as the version that comes with NixOS 24.05 is an older version that has all sorts of problems
Here is how I did it
hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.mkDriver {
version = "555.58";
sha256_64bit = "sha256-bXvcXkg2kQZuCNKRZM5QoTaTjF4l2TtrsKUvyicj5ew=";
sha256_aarch64 = "sha256-7XswQwW1iFP4ji5mbRQ6PVEhD4SGWpjUJe1o8zoXYRE=";
openSha256 = "sha256-hEAmFISMuXm8tbsrB+WiUcEFuSGRNZ37aKWvf0WJ2/c=";
settingsSha256 = "sha256-vWnrXlBCb3K5uVkDFmJDVq51wrCoqgPF03lSjZOuU8M=";
persistencedSha256 = "sha256-lyYxDuGDTMdGxX3CaiWUh1IQuQlkI2hPEs5LI20vEVw=";
};
We found the solution in this other thread I posted in the NixOS forums: