This is great advice. Another related issue I see is when you engineer a new feature that can't always be 100% accurate because the source data is spotty but you intuitively think the new festure should help the classifier anyway when it is present. And if the new feature's feature importance in the trained model turns out really high, you think you've done something great. But in the end you made model that simply detects the presence of your new feature which you knew wasn't 100% accurate anyway because the source data it is derived from is spotty. So you've accomplished precisely nothing.

There was an article on here a few months back that said something like, "The majority of today's applications of AI could be just as well - if not better - served by a simple heuristic"

I think this is minor, but I noticed the example doesn't use error bars for the customer numbers. The customer counts in the product/churn categories are counting statistics, so have an associated Poisson uncertainty. My guess is that considering the uncertainties when doing the likelihood parameter estimation won't fully obviate this issue, but I wonder how much it would help. I'm also not sure if real-world implementations commonly consider such uncertainties on their measured metrics.