Winter jujube has thin skin and crisp meat. It is rich in vitamin C and minerals and is deeply loved by consumers. However, there are many kinds of diseases of winter jujube, which seriously restricts the industrialization development of winter jujube. In order to overcome this problem, a winter jujube fruit disease recognition method is proposed based on hierarchical convolutional neural network（HCNN）. HCNN consists of three convolutional neural network（CNN） models with the same structure and one SVM classifier. Firstly, RGB, HIS and Lab images of the original winter fruit disease image are respectively input into three CNNs of HCNN. Then, the feature maps of three CNN models are integrated into a feature vector at the classification layer of HCNN. Finally, the SVM classifier is used to classify the disease images. This method can automatically extract the effective features from the disease image of winter jujube instead of the artificial design features. A series of experiments are carried out on the fruit disease image dataset. The experimental results show that the method can realize accurate, stable and efficient identification of winter jujube fruit disease types by making full use of the features of different colors in the image,and provide reference for the development of winter jujube fruit disease control system.