Visible + NIR information for CNN face recognition
What if we made the camera sensitive to near-infrared to better recognise faces? I built an original VNIR dataset (52 identities, 3 poses), trained two CNN architectures, and compared their performance against plain RGB. The result surprised me: 3-channel full-spectrum images beat 4-channel RGB + I.
The NIR signal we silently throw away
The CCD/CMOS sensors in every digital camera are sensitive both to visible light (400-700 nm) AND near-infrared (700-1200 nm). But an internal filter — the ICF (Infrared Cut-off Filter) — blocks IR so that colours look natural to the human eye.
Remove that filter and you recover NIR information 'for free', mixed into the three RGB channels. Skin, eyes and bone respond differently in NIR: skin looks more uniform (fewer surface defects show), sub-cutaneous structures become visible. Can we leverage that signal to recognise a face more robustly?
Three images per pose, one camera
Capture: 52 identities · 3 poses · with / without glasses · 2048 × 1536 JPEG
The setup, the dataset, the two CNNs
Hardware hack: SONY F717 with a 'night shot' mode whose ICF can be mechanically retracted. Tripod + 4 IR LEDs to compensate for the lack of ambient NIR indoors. A Kodak Wratten 87 filter mounted in front of the lens for pure-NIR captures.
- Original VNIR dataset: 52 identities, 3 poses (front / quarter-right / quarter-left), with and without glasses for those who wear them.
- 3 versions per pose: plain RGB (ICF on), full-spectrum (ICF off, stored as RGB by the camera), pure NIR (Wratten 87 filter, green channel extracted as monochrome).
- Resolution: 2048 × 1536 JPEG, high-resolution crop on the face region.
- Two CNN architectures compared: (a) classic 3-channel CNN trained on RGB, tested also on full-spectrum; (b) 4-channel CNN R, G, B, I with I = pure NIR image.
What works, what does not
Full-spectrum 3 channels: a surprise
A CNN trained on plain RGB generalises well to full-spectrum images, without any retraining. The visible + NIR mixture in the three channels does not confuse recognition — accuracy goes up.
RGB + I 4 channels: a disappointment
Adding NIR as a separate fourth channel degrades performance compared to the 3-channel full-spectrum approach. The architecture cannot exploit the isolated channel as well as it does NIR 'baked' into RGB.
The blue channel, biggest winner
When NIR is injected into the RGB channels, the blue channel improves the most. Blue is usually the weakest signal (sensors are less sensitive there), so the signal-to-noise ratio benefits the most from the NIR boost.
Lighting robustness
Full-spectrum is more stable than plain RGB under varying indoor lighting — ambient NIR fills in for the visible signal as light dims.
What this publication brings
- First public VNIR dataset dedicated to face recognition. Existing bases (e.g. CASIA NIR-VIS 2.0) contain RGB and NIR separately, not full-spectrum.
- Empirical demonstration that a custom multi-channel architecture is not required to leverage NIR: a plain CNN on full-spectrum is enough — and actually does better.
- Reproducible capture methodology: consumer camera + IR LEDs + filter. No lab-grade equipment needed.
- Practical recommendation for face-recognition systems exposed to variable lighting: remove the ICF rather than adding a separate NIR sensor.
Venue & context
- Published at IEEE EUVIP 2018 (European Workshop on Visual Information Processing).
- Work carried out during my research master's at MIA Laboratory · University of La Rochelle.
- Keywords: CNN, face recognition, NIR imaging, multispectral, full-spectrum, dataset.