I recently had this question in consulting:
I’ve got 12 out of 645 cases with Mahalanobis’s Distances above the critical value, so I removed them and reran the analysis, only to find that another 10 cases were now outside the value. I removed these, and another 10 appeared, and so on until I have removed over 100 cases from my analysis! Surely this can’t be right!?! Do you know any way around this? It is really slowing down my analysis and I have no idea how to sort this out!!
And this was my response:
I wrote an article about dropping outliers. As you’ll see, you can’t just drop outliers without a REALLY good reason. Being influential is not in itself a good enough reason to drop data.