In clinical practice, the diagnosis of Prostate cancer (PCa) and the treatment decision are based on Gleason scores and nomograms. However, Gleason scores suffer from high inter-observer variability and nomograms are only able to model linear dependencies between patient parameters. Therefore, the goal of our research is to provide individual and objective prognoses for PCa patients through predicting relapse after radical prostatectomy (RPE). To this end we build and train deep learning-based models that predict the probability of a patient having a relapse from either H&E-stained images of biopsies after the RPE or electronic health records that reflect the patient’s clinical history related to prostate cancer.