目的 H.264/AVC帧间预测编码需要对所有可能编码模式计算并比较率失真代价，众多的模式类型导致了P帧编码的计算复杂度非常高。为此提出一种针对P帧的基于决策树的快速候选模式选择算法。方法 在对宏块进行16×16的帧间运动估计后，首先根据残差宏块中4×4全零系数块个数对部分宏块直接选择出候选模式；然后使用16个4×4块的变换域系数绝对值之和（SATD）值，采用决策树分类方法对其余宏块选择候选模式。结果 由于只需对候选模式进行编码，因此有效降低了编码器的计算复杂度。实验结果表明，与原始全搜索编码算法相比，该算法对不同运动程度的视频序列获得了较一致的编码时间的节省，同时平均峰值信噪比的损失和平均比特率的增加均较少。结论 新的P帧帧间预测候选模式选择算法，根据帧间运动估计后的残差宏块信息，采用决策树方法对候选模式集进行分类。实验结果表明，该算法能在保证视频编码质量的前提下，有效地降低编码过程中的计算量，缩短编码时间。

Objective H.264/AVC encoding standard effectively improves coding efficiency by using the intra and inter prediction techniques.However, due to the use of variable block sizes and rate-distortion optimization techniques, there is an extremely high computational complexity for inter-prediction in H.264/AVC, which limits the applications of H.264/AVC.This is because the encoder needs to exhaustively check all the prediction modes for identifying the best one.Therefore, reducing the complexity of the encoder is very important especially for real-time applications.Choosing a small number of candidate modes other than all the modes to perform rate-distortion optimization can reduce the complexity and speed-up the encoding process.In this paper, a fast intermode decision algorithm for P-frame encoding based on the decision tree is proposed to reduce the number of modes required to be checked.Method The inter frame motion estimation for a 16×16 size is firstly performed when a macroblock (MB) of P-frame is encoded by using the inter prediction technique.Additionally, we use the information of residual MB after motion compensation to select a small number of candidate modes from all possible prediction modes to calculate and compare the rate-distortion cost.At first, the number of 4×4 all-zero blocks in the residual MB can describe how accurate the 16×16 inter prediction is.Thus with the statistical analysis of the correlation between the number of 4×4 all-zero blocks in the residual MB and the best prediction mode in full search algorithm, the candidate modes of some MBs can be directly determined based on the number of all-zero blocks in the residual MB.Then, for the remaining MBs, machine-learning tools are used to exploit the correlation between the residual MB information and the best mode in H.264/AVC full search algorithm.By using the decision tree classification algorithm, candidate modes can be selected based on the SATD features of residual MB.Result The proposed algorithm efficiently reduces the number of candidate modes for P-frame inter-prediction in the H.264/AVC encoder.Experimental results show that the proposed algorithm can achieve significant encoding time saving for all the test video sequences covering a wide range of motion activities. Meanwhile there is the less PSNR degradation and a small bit rate increase, compared with the reference full search algorithm.Although the time-saving of the proposed algorithm is not better than those of the comparative algorithms, the time-savings for video sequences with different motion degrees are almost identical and the rate distortion performance is obvious better.Conclusion H.264/AVC video coding standard effectively improves coding efficiency but at the cost of high computational complexity.This paper presents a fast and efficient intermode decision algorithm to speed up the encoding process.The algorithm reduces the computational complexity by selecting a small number of candidate modes from the set of all possible prediction modes.We use the decision tree classification algorithm to determine candidate modes based on the information in residual MB after 16×16 inter motion estimation.The experimental results indicate that our algorithm reduces the computational complexity with a little loss in PSNR and increment in the total bit rate.