Patient embedding representation is a crucial task for the healthcare domain to predict patients’ future diseases, future inpatients visits, or medical status deterioration. This representation can be done using EHR data containing diagnoses, procedures, and medication codes. This lecture will present our patient representation models, from static code representation using word2vec to contextual embedding using transformers. We adapted the BERT framework originally developed for the text-domain to the structured EHR domain. Our pretrained EHR language model is a contextualized embedding model trained on patients sequences on masked language model task. Then we fine-tune the model for the downstream prediction task. Our results show that this patient embedding technique substantially improves the prediction accuracy of our models.