Ask HN: How do you manage SSH known_hosts?
In our company we are managing a few dozen virtual machines. A lot is automated, but we still need to manually ssh in to machines often.
How do you manage your ~/.ssh/known_hosts file, to make sure you're not connecting to a malicious host? We spin up machines often, and it's not feasible to manually check every new host key. Additionally, DNS entries are often updated to new hosts, which produces nasty "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" messages, which some people also learn to ignore after a while.
Are there any tools to keep a company-wide known_hosts file in sync? Or should we not be using SSH directly to all these machines at all?
That's a very good question, and one that I haven't seen answered in a satisfactory way. People rarely think about this particular issue, much less try to solve it. It's one of the reasons I consider SSH unsuitable for unattended batch communication between servers.
What I would try is running just after installing a machine a registration script that reports server's public SSH keys. Then it would be quite easy to generate a system-wide known_hosts file on a server that would be used as a jump host (sometimes it's called "bastion host"), or distribute it in any other way.
Note that Puppet with its most popular deployment method, with master node, uses registration process and reports SSH keys (and other information) back to the master.