This paper analyses the design choices of face detection architecture thatimprove efficiency between computation cost and accuracy. Specifically, were-examine the effectiveness of the standard convolutional block as alightweight backbone architecture on face detection. Unlike the currenttendency of lightweight architecture design, which heavily utilizes depthwiseseparable convolution layers, we show that heavily channel-pruned standardconvolution layer can achieve better accuracy and inference speed when using asimilar parameter size. This observation is supported by the analysesconcerning the characteristics of the target data domain, face. Based on ourobservation, we propose to employ ResNet with a highly reduced channel, whichsurprisingly allows high efficiency compared to other mobile-friendly networks(e.g., MobileNet-V1,-V2,-V3). From the extensive experiments, we show that theproposed backbone can replace that of the state-of-the-art face detector with afaster inference speed. Also, we further propose a new feature aggregationmethod maximizing the detection performance. Our proposed detector EResFDobtained 80.4% mAP on WIDER FACE Hard subset which only takes 37.7 ms for VGAimage inference in on CPU. Code will be available athttps://github.com/clovaai/EResFD.