Background: Histopathological images of colorectal liver metastasis (CRLM) contain rich morphometric information that may predict patient outcomes, but current indicators depend on labor-intensive and subjective visual estimation. Herein, we aimed to develop an automated framework for tissue classification of routine H&E stained whole-slide images and establish a risk-scoring model for better prognosis prediction in patients with CRLM. Methods: Using 161,371 hand-delineated image patches, we trained a robust deep convolutional neural network- CRLM-SPA for accurate classification of CRLM into seven tissue types. With the tissue classification results on two independent in-house cohorts (SYSUCC: n = 433, BJCH: n = 404), we systematically quantified spatial organization features (SOFs), involving whole-slide, tumor-infiltrating and tumor-distal SOFs for different tissue types as well as the interactions between tumor and non-tumor tissues. Subsequently, univariate Cox proportional hazards regression analysis was performed to investigate the association between various SOFs and patient outcomes. Nonredundant SOFs that are clinically relevant were selected to build a risk-scoring model for prognosis using multivariate Cox regression analysis in the SYSUCC cohort, followed by validation in the BJCH cohort. Results: CRLM-SPA achieved an overall classification accuracy of > 93% in an independent set of 17,653 image patches. With the classification result, we calculated various SOFs and built a four-SOF risk-scoring model that significantly predicted overall survival (OS) in the discovery cohort, SYSUCC ( P = 1.40 10^-5^; HR = 2.26; 95%CI: 1.55 - 3.29) and the independent validation cohort, BJCH ( P = 4.59 × 10^-4^; HR = 2.00; 95%CI: 1.35 - 2.97). The prognostic performance of our SOF risk-scoring model is independent of the clinical risk score (CRS) system. Further stratification analyses in patients without preoperative chemotherapy (CTx) indicated that adjuvant CTx consistently improved OS in the SOF high-risk subgroups in both cohorts ( P = 0.018 and 0.049), but not in the SOF low-risk subgroups. Furthermore, a combined scoring system incorporating SOF and CRS considerably improved the prognostic performance in both cohorts over the individual SOF and CRS systems. Conclusions: CRLM-SPA showed a high accuracy in tissue classification and robustness in extracting prognostic information from H&E images. The SOF risk-scoring system demonstrated a strong and robust prognostic value that is independent of CRS, and could therefore provide a time- and cost-efficient tool to assist clinical decision making for patients with CRLM.