Modern vehicles are standardly equipped with interior cameras that can capture both visual and (near infrared) NIR modalities. Currently, they are installed to monitor the driver and thereby prevent accidents. In addition, there is also the possibility of using them to estimate the vital functions of the occupants.

In the future, the vital sign estimation could detect critical events such as sudden heart attacks, strokes or fatigue at an early stage. It can then also detect whether drivers or passengers are impaired, e.g. under the influence of drugs or alcohol.

This is particularly useful if the intelligent vehicle prevents an impaired driver from driving the vehicle. This significantly reduces the risk of a fatal accident.

A modern car analyses the vital signs and can assess whether the driver is in a phase of drowsiness where micro sleep is very likely. The camera can use the algorithm to detect changes in heart rate and breathing that typically occur during this phase. The system concludes how likely it is that the phenomenon of micro sleep may occur in the near future.

It is precisely in this phase that intelligent assistance systems such as lane departure warning systems can be useful in reducing the risk of accidents caused by micro sleep.

Many people stay in their cars often and for long periods of time without moving much. This allows smart vehicles to record and monitor their vital parameters, or changes in heart rate and breathing, over the long term and with greater accuracy than when they are moving. In contrast to a smartwatch, which you move a lot in everyday life, the prediction is much more accurate when you are at rest.

This can be used in preventive medicine. Changes in breathing as well as heart rate allow conclusions to be drawn about the current state of fitness and can predict the risk of developing cardiovascular diseases such as cardiac arrhythmias, heart attacks and asthma.

The vehicle has the potential to act as a substitute for a doctor’s traditional waiting room, for example, when the patient waits in their car while their current and long-term vital signs are transmitted to the practice via camera. The doctor can then decide whether an in-person appointment is necessary, prescribe medication or adjust medication.

While it is unlikely that smart vehicles will completely replace doctor’s visits in the future, it is quite possible that long-term monitoring of vital signs could minimise many in-person visits.

Vital parameter estimation can also help to increase the well-being of the occupants by inferring emotions and activating comfort functions in the vehicle depending on the situation.

This has many advantages over the algorithms used so far, which focus exclusively on facial expressions. This is because they cannot reliably estimate emotions if people wear a beard or are skilled at hiding their emotions.

Vital signs are different. They distinguish the emotional state more reliably from the behavior of heart rate and breathing, and estimate fairly accurately whether people seem focused or are angry.

To estimate heart rate, IAV uses specific regions of the face that are usually uncovered and criss-crossed with numerous arteries and veins. This allows Deep Learning to use a camera pointed at the face to capture subtle colour and brightness variations that occur when the arteries pulsate, causing the blood volume to change.

More specifically, it works via the photoplethysmographic signal (PPG). Arterial blood absorbs light more efficiently than surrounding tissue, and when the arteries expand due to the pulse, the absorption rate of the light changes. The camera detects this change and maps it to a PPG signal.

In order to estimate the respiratory rate, IAV used the optical flow method. Here, an optical flow algorithm detects chest movements in the video images and assigns them to a specific respiratory rate. The respiration rate is derived from the movement of the thorax and upper abdomen during the inhalation and exhalation phases. In this way, this movement can be assigned to a respiratory signal, which is recognised as a person’s respiratory rate.

Patrick Laufer will again take part as a speaker at the InCabin exhibition on 21.06.2023. In his presentation, he will discuss whether a single RGB NIR camera is sufficient to determine the heart and breathing rates of all occupants in a vehicle. He will also highlight possible problem cases.

Are you interested in this topic? Then also read the article on camera-based estimation of height, weight and gender of vehicle occupants.

For more information, please contact Patrick Laufer on LinkedIn or at patrick.laufer@iav.de.