Recently, the relationship between emotional arousal and depression has been studied. Focusing on this relationship, we first developed an arousal level voice index (ALVI) to measure arousal levels using the Interactive Emotional Dyadic Motion Capture database. Then, we calculated ALVI from the voices of depressed patients from two hospitals (Ginza Taimei Clinic [GTC] and National Defense Medical College hospital [NDMC]) and compared them with the severity of depression as measured by the Hamilton Rating Scale for Depression (HAM-D). Depending on the HAM-D score, the datasets were classified into a no depression (HAM-D<8) and a depression group (HAM-D≥8) for each hospital. A comparison of the mean ALVI between the groups was performed using the Wilcoxon rank-sum test and a significant difference at the level of 10% (p = 0.094) at GTC and 1% (p = 0.0038) at NDMC was determined. The area under the curve (AUC) of the receiver operating characteristic was 0.66 when categorizing between the two groups for GTC, and the AUC for NDMC was 0.70. The relationship between arousal level and depression severity was indirectly suggested via ALVI.