The Library
Robust multiview multimodal driver monitoring system using masked multi-head self-attention
Tools
Ma, Yiming, Sanchez Silva, Victor, Nikan, Soodeh, Upadhyay, Devesh, Atote, Bhushan and Guha, Tanaya (2023) Robust multiview multimodal driver monitoring system using masked multi-head self-attention. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, Canada, 18-22 Jun 2023. Published in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 2617-2625. ISBN 9798350302493. doi:10.1109/CVPRW59228.2023.00260 ISSN 2160-7516.
|
PDF
WRAP- robust-multiview-multimodal-driver-monitoring-system-Ma-2023.pdf - Accepted Version - Requires a PDF viewer. Download (3817Kb) | Preview |
Official URL: https://doi.org/10.1109/CVPRW59228.2023.00260
Abstract
Driver Monitoring Systems (DMSs) are crucial for safe hand-over actions in Level-2+ self-driving vehicles. State-of-the-art DMSs leverage multiple sensors mounted at different locations to monitor the driver and the vehicle’s interior scene and employ decision-level fusion to integrate these heterogenous data. However, this fusion method may not fully utilize the complementarity of different data sources and may overlook their relative importance. To address these limitations, we propose a novel multi-view multimodal driver monitoring system based on feature-level fusion through multi-head self-attention (MHSA). We demonstrate its effectiveness by comparing it against four alternative fusion strategies (Sum, Conv, SE, and AFF). We also present a novel GPU-friendly supervised contrastive learning framework SuMoCo to learn better representations. Furthermore, We fine-grained the test split of the DAD dataset to enable the multi-class recognition of drivers’ activities. Experiments on this enhanced database demonstrate that 1) the proposed MHSA-based fusion method (AUC-ROC: 97.0%) outperforms all baselines and previous approaches, and 2) training MHSA with patch masking can improve its robustness against modality/view collapses. The code and annotations are publicly available 1 .
Item Type: | Conference Item (Paper) | ||||||
---|---|---|---|---|---|---|---|
Alternative Title: | |||||||
Subjects: | T Technology > TA Engineering (General). Civil engineering (General) T Technology > TE Highway engineering. Roads and pavements T Technology > TK Electrical engineering. Electronics Nuclear engineering T Technology > TL Motor vehicles. Aeronautics. Astronautics |
||||||
Divisions: | Faculty of Science, Engineering and Medicine > Science > Computer Science | ||||||
Library of Congress Subject Headings (LCSH): | Automated vehicles, Automated vehicles -- Technological innovations, Pattern recognition systems , Computer vision , Intelligent transportation systems | ||||||
Journal or Publication Title: | Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) | ||||||
Publisher: | IEEE | ||||||
ISBN: | 9798350302493 | ||||||
ISSN: | 2160-7516 | ||||||
Official Date: | 2023 | ||||||
Dates: |
|
||||||
Page Range: | pp. 2617-2625 | ||||||
DOI: | 10.1109/CVPRW59228.2023.00260 | ||||||
Status: | Peer Reviewed | ||||||
Publication Status: | Published | ||||||
Re-use Statement: | © 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. | ||||||
Access rights to Published version: | Restricted or Subscription Access | ||||||
Date of first compliant deposit: | 18 May 2023 | ||||||
Date of first compliant Open Access: | 19 May 2023 | ||||||
Conference Paper Type: | Paper | ||||||
Title of Event: | IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) | ||||||
Type of Event: | Conference | ||||||
Location of Event: | Vancouver, Canada | ||||||
Date(s) of Event: | 18-22 Jun 2023 | ||||||
Related URLs: | |||||||
Open Access Version: |
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |
Downloads
Downloads per month over past year