Sustainable development of small towns with agricultural characteristics is an important guarantee to realize the strategic goal of rural revitalization. In order to effectively promote the sustainable development of small towns with agricultural characteristics, 23 factors affecting the sustainable development of small towns with agricultural characteristics were summarized in the paper through the analysis of the development model of small towns with agricultural characteristics, combined with existed researches and on site investigation. Factors affecting the sustainable development of small towns with agricultural characteristics were explained and summarized using factor analysis. The results showed that the factors affecting the sustainable development of small towns with agricultural characteristics could be summarized into four aspects: industrial driving factors, talent driving factors, institutional driving factors and ecological driving factors. In the context of the strategy of rural revitalization, corresponding strategies for the sustainable development of small towns with agricultural characteristics were put forward in view of these factors.