Background: Cancer associated fibroblast (CAF) in tumor microenvironment is associated with poor prognosis and chemo-resistance in multiple solid tumors, however, there is lack of universal measure of CAF in colorectal cancer (CRC). The aim of this study was to assess fibroblast-signature for predicting outcome and analyze relevant mechanism. Methods: A dataset including 316 CRC patients without adjuvant chemotherapy was used as the discovery cohort for the identification of prognostic fibroblast-related genes (FRGs). A total of 1,352 CRC patients were then divided into one training cohort (n = 461) and two validation cohorts (n = 338, n = 553, respectively) for the construction of fibroblast-related gene signature (FRGS) and the verification of its prognostic value in stage II/III CRC patients. Functional annotation and analysis were performed to reveal the relevant mechanism. Results: A 11-gene signature was derived, which was prognostic for stage II/III CRC patients in two validation cohort (Validation-1 cohort: HR = 1.90, 95%CI = 1.16-3.12, P< 0.01; Validation-2 cohort: HR = 1.95, 95%CI = 1.39-2.73, P< 0.001). High CAF risk was correlated with worse prognosis in CRC patients without adjuvant chemotherapy (HR = 3.63, 95%CI = 2.24-5.88, P< 0.001), but not in patients who received adjuvant chemotherapy ( P= 0.154). Similar trends were found in Validation-1 cohort. After integrated with clinical characteristics, FRGS was confirmed as an independent prognostic factor after adjusted for TNM stage of tumor in multivariate analysis (Training cohort: HR = 3.19, 95%CI = 1.88-5.41, P< 0.001; Validation-1 cohort: HR = 5.00, 95%CI = 1.58-15.85, P= 0.007; Validation-2 cohort: HR = 2.99, 95%CI = 1.44-6.21, P= 0.003). Furthermore, enrichment analysis found that anti-tumor immune response was suppressed in the high CAF risk group. Conclusions: The 11-gene FRGS had independent prognostic value for CRC patients, as well as in prediction of benefit from chemotherapy. CAF in tumor microenvironment might impact on the prognosis of CRC patients via inhibiting immune response.