“Seeing” Electric Network Frequency from Events

Abstract

Most of the artificial lights fluctuate in response to the grid's alternating current and exhibit subtle variations in terms of both intensity and spectrum, providing the potential to estimate the Electric Network Frequency (ENF) from conventional frame-based videos. Nevertheless, the performance of Video-based ENF (V-ENF) estimation largely relies on the imaging quality and thus may suffer from significant interference caused by non-ideal sampling, motion, and extreme lighting conditions.

In this paper, we show that the ENF can be extracted without the above limitations from a new modality provided by the so-called event camera, a neuromorphic sensor that encodes the light intensity variations and asynchronously emits events with extremely high temporal resolution and high dynamic range. Specifically, we first formulate and validate the physical mechanism for the ENF captured in events, and then propose a simple yet robust Event-based ENF (E-ENF) estimation method through mode filtering and harmonic enhancement.

Furthermore, we build an Event-Video ENF Dataset (EV-ENFD) that records both events and videos in diverse scenes. Extensive experiments on EV-ENFD demonstrate that our proposed E-ENF method can extract more accurate ENF traces, outperforming the conventional V-ENF by a large margin, especially in challenging environments with object motions and extreme lighting conditions.

Dataset

Event-Video ENF Dataset (EV-ENFD)

We construct the EV-ENFD for real illumination scenes using two cameras with a rolling shutter mechanism to acquire videos (Video-LR and Video-HR) and an event camera to acquire events. All cameras are fixed on a tripod to ensure synchronized recording time and scene.

The EV-ENFD has three categories according to the capturing conditions: static for static scenes without relative motions, dynamic for dynamic scenes caused by object or camera motions, and extreme lighting for the scenes with over or under-exposed regions.

Static Scenes

Dynamic Scenes (Object Motion, Camera Motion, and Both)

Extreme Lighting

Comparison

Results on Static Scene

Results on Dynamic Scene

Results on Extreme Lighting

References

[1] Ravi Garg, Avinash L. Varna, Adi Hajj-Ahmad, and Min Wu. “Seeing” ENF: Power-Signature-Based Timestamp for Digital Multimedia via Optical Sensing and Signal Processing. IEEE Transactions on Information Forensics and Security, 8(9):1417–1432, 2013.
[2] Saffet Vatansever, Ahmet Emir Dirik, and Nasir Memon. Analysis of rolling shutter effect on ENF-based video forensics. IEEE Transactions on Information Forensics and Security, 14(9):2262–2275, 2019.
[3] Guang Hua, Han Liao, Haijian Zhang, Dengpan Ye, and Jiayi Ma. Robust ENF Estimation Based on Harmonic Enhancement and Maximum Weight Clique. IEEE Transactions on Information Forensics and Security, 16:3874–3887, 2021.
[4] Mark Sheinin, Yoav Y Schechner, and Kiriakos N Kutulakos. Computational Imaging on the Electric Grid. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6437–6446, 2017.
[5] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154–180, 2020.
[6] Wei Liao, Xiang Zhang, Lei Yu, Shijie Lin, Wen Yang, and Ning Qiao. Synthetic Aperture Imaging With Events and Frames. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17735– 17744, 2022.

BibTeX

@inproceedings{xu2023seeing,
  title={"Seeing" Electric Network Frequency From Events},
  author={Xu, Lexuan and Hua, Guang and Zhang, Haijian and Yu, Lei and Qiao, Ning},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={18022--18031},
  year={2023}}