Chatter in the process of end milling would lead to poor surface finish, low material removal rate, severe tool wearand noisy workplace, and etc.Chatter vibration can be detected by most chatter detection system, however, it already has serous effects on the surface quality of workpieces as well as the cutting tools when it occurring.Therefore, chatter detection system must find chatter characteristicsin the early state.The nonlinear machining process would cause complex frequency componentsin the vibration signal,so it was difficult to obtain reliable chatter feature using a single timefrequency analysis method.Wavelet packet decomposition was employed to define the chatter emerging frequency range, and it was reconstructed.The cepstrum of chatter emerging frequency range wasusedto identify the stable, transition and chatter states. The study results show that the method can accurately distinguish the stable, transition and chatter states in end milling processes.