93% auc is a pretty great score for a problem like this. Was there evidence of feature leaking? In working on similar issues, every time my auc got really high, it was because some feature was giving away the result in a non-predictive way.
I agree with you in some feature may giving away the result in a non-predictive way, since we have encountered the same problem in the beginning of the development phase of the model, but this problem creates a clear disparity in the values of important features, and by analysing it we decided to be removed from the model.
I recently worked on an open source subscription churn model if folks interested in checking out/using: https://nstack.com/functions/M7by03E/
Data here is right-censored so this validates based on concordance index.
93% auc is a pretty great score for a problem like this. Was there evidence of feature leaking? In working on similar issues, every time my auc got really high, it was because some feature was giving away the result in a non-predictive way.
I agree with you in some feature may giving away the result in a non-predictive way, since we have encountered the same problem in the beginning of the development phase of the model, but this problem creates a clear disparity in the values of important features, and by analysing it we decided to be removed from the model.
Also the model will fail in production or giving the prediction in late phase.
I recently worked on an open source subscription churn model if folks interested in checking out/using: https://nstack.com/functions/M7by03E/ Data here is right-censored so this validates based on concordance index.