To address the challenge of limited training data for machine learning models in the healthcare domain, we advocate for human-in-the-loop machine learning, which involves domain experts in an inter- active process of developing predictive models. Interpretability offers a promising way to facilitate this interaction. We describe an approach that offers a simple decision tree interpretation for any complex blackbox machine learning model. In a case study with physicians, we find that they were able to use the interpretation to discover an unexpected causal issue in a personalized patient risk score trained on electronic medical record data. To account for dynamics in disease progression, we advocate for building decision models that integrate predictions of the disease progression at the individual patient level with system models capturing the dynamic operational environments. We describe a case study on hospital inpatient management, showing how to build a Markov decision framework that leverages predictive analytics on patient readmission risk and prescribes the optimal set of patients to be discharged each day.