q1) I am aware of other ways to extract out “variable importance ranking” – please share links or describe other ways you use. What are pro’s and con’s? Some approaches are algorithm specific, some are algorithm agnostic.
q2) How do you provide “record or person level reason codes” when needed by the application? I am aware of score card modeling for linear methods. How would you deal with non-linear models, with ensembles of models?
q3) How do you manage the lifecycle of models that you have had in production? Do they get replaced after a fixed time (i.e. a day for banner ads, a month, a year, …?). Or if you use data driven metrics, what metrics do you use? Do you use a strict policy or threshold, or allow some leeway, based on available resources on when to start refreshing a model?
q4) When replacing a model, do you always fully replace it? Do you ever just do a minimal retrain on fresher data, for a “model refresh”? Do you have any other choices for how you update models?