Accurate image registration is essential for eye movement compensation in optical coherence tomography (OCT) and OCT angiography (OCTA). The spatial resolution of an OCT instrument is typically anisotropic, i.e., different resolutions in the lateral and axial dimensions. When OCT images have anisotropic pixel resolution, residual distortion (RD) and false translation (FT) were always observed after image registration for rotational movement. In this study, RD and FT were quantitively analyzed over different degrees of rotational movement and various lateral and axial pixel resolution ratio (RL/RA) values. The theoretical analysis confirmed that the RD and FT increase significantly with the rotation degree and RL/RA. An image resizing assisting registration (RAR) strategy was proposed for accurate image registration. The performance of direct registration (DR) and RAR for retinal OCT and OCTA images were quantitatively compared. Experimental results confirmed that unnormalized RL/RA causes RD and FT, RAR can effectively improve the performance of OCT and OCTA image registration and distortion compensation.