Why You Should Be Using Git Worktrees
Evidence is showing that Continiuos Intergration/Continious Development is the most effective development method. (See dora.dev) Working without branches (on the server) is the best way to implement this. (Search 'ci branches' on YouTube.)
Here's a quick blog post I wrote about the merit of git worktrees. Any thoughts or feedback let me know!
> However, since worktrees are actually completely separate directories, you do have to install all your dependencies in each new worktree.
What we need here is package caching in order to reduce download bandwidth and time, and its deduplication to avoid consuming storage space. This is important because worktrees are much more likely to share the same dependencies than unrelated projects. There may be language/toolchain specific solutions for this.
In case of nodejs/frontend projects, both npm and pnpm implement download caching. However, npm seems to copy the relevant modules to the project directory, duplicating the files in storage. Pnpm[1] on the other hand, uses a global cache and symlinks/hardlinks the relevant version of the relevant module to the project tree. This will save a lot of space when multiple worktrees are used.
In the python ecosystem, only conda[2] seems to provide this ability.
The sccache project [3] provides similar advantages to C, C++, Rust and CUDA projects. It caches the binary artifacts of compilation, so that can be used to speed up even first compilation of projects. By default, these caches can be shared between different projects on the same system. But with networked storage backends, sccache can be used to share build artifacts across a network and even enable parallel builds on multiple builds. (Added: Cargo seems to globally cache sources and deal with it on its own without using any links at all.)
Another important detail that might be worth consideration is the filesystem on which the worktrees reside. Copy-on-write filesystems like ZFS, btrfs and bcachefs provide automatic deduplication when copying files within a subvolume. I'm not sure what happens when worktrees are created by git. But they at least have tools to do manual deduplication (btrfs does). Perhaps we can just copy the package manager and build directories (like node_modules and build) from another worktree before running the development tools. Those tools are designed to handle stale cache anyway. So the wrong files wont be much of an issue for them. Perhaps tools like npm should look into making use of such filesystems, because you can get the same advantage as pnpm with practically no extra effort.
[1] https://pnpm.io/motivation#saving-disk-space
[2] https://docs.conda.io/projects/conda/en/latest/user-guide/co...