I feel like the primary use case for such a technology is manipulating and profiling people over video chat, maybe even autonomously. Hiring managers, HR, landlords, and police are obvious customers.
This tech (detecting pulse from regular video) has been around almost 20 years now, and this doesn't seemed to have happened yet.
You see this type of thing in spy movies, but I'm not sure it's that useful in real life. You're basically taking one piece of data a polygraph uses, but without the most important component (skin conductance). Polygraph accuracy isn't that great to begin with. You can profile and manipulate people more effectively based on their reactions and behaviour, and their pulse will be much harder to interpret.
It's probably not great for this. Resting heart rate is correlated with fitness, but candidates can be nervous during interviews which can drastically change their heart rate. You'd probably be able to make better guesses about their health through looking at them (do they look sickly or overweight?) and interviewing them.
It's also not a specific enough signal for the types of health conditions that require taking time off work. Assuming they could get an accurate resting heart rate, most of the time it'll just indicate whether the candidate does a lot of exercise.
You are talking about healthy and borderline. I am talking about unhealthy. If someone has heart rate of say 150 they are far more likely to have other issues.
I don't know any commercial uses of such tech today. I'm not saying they don't exist. I just don't know of them.
I had said I don't think it's very useful for "manipulating and profiling people over video chat", so I wouldn't really expect there to be a commercial product for that. Probably it's used in fitness or heart rate monitoring apps for people that don't have a fitness tracker device and prefer not to manually count their pulse.
The core algorithm is really simple. You find a patch of skin. Take the average color of the pixels in that patch. The color will become more reddish each pulse. Do an FFT and take the strongest peak in the plausible heart rate range. You could prototype this in a few hundred lines of python.
If this were useful for police or hiring managers, someone could have use the tech to make an app for them within the past 19 years.
Of course, companies have a history of trying to market a lot of BS metrics (e.g. graphology, MBTI) to hiring managers, so I wouldn't be that surprised to see a company claim they can predict employee success using pulse. Whether it works is another story.
Liveness detection of what appears to be a person in video, but is a photo, a skulpture, or some other attempt to falsely identify to an identity system as another.
I don't it's ever been practical to ship in a product? You need ~20 seconds of data to stabilise the reading, and any large motion ruins it - even though Microsoft Research demonstrated a Kinect could detect heartrate in a lab setting, it wasn't viable to ship in a fitness game.
I'd expect the primary use case to be liveness detection, to validate that the person the facial recognition identifies is not a photo, a sculpture, person wearing facial prosthetics, or a mask. I've coding such software for that exact purpose.
I would prefer some kind of privacy statement or even some kind of explanation about what is going on before I just randomly turn my webcam on. This might be great and I’m proud of you for launching but I don’t do things like that. Heck, videos can make a person’s heart race - I had my first attack at 39 and that’s a hell of a lot of risk.
I haven't dug deeper due to time availability, but for the same sake of privacy, I've found:
1. `/api/event` endpoint mentioned in the `/stats/script.js` file;
2. There's `/parties/lobby/main/telemetry` in a minified JavaScript chunk asset;
3. There's VitalLens mentioned, and there's an error string in the same asset: "A valid API key or proxy URL is required to use VitalLens. If you signed up recently, please try again in a minute to allow your API key to become active. Otherwise, head to https://www.rouast.com/api to get a free API key."
It’s not very accurate. Maybe because of the camera fidelity. It was about 10bpm lower than actual for me. Seems to operate off of subtle motions caused by pulse. It was even worse at detecting breathing.
Yeah, I couldn't be bothered getting my accurate chest strap out, but my watch (which is generally very close to the strap) was anywhere from 10-20 off what it was reporting. This is sitting down, 30min after a run.
Love the attempt, but after giving it a good image and letting it run for a while it's totally off. I measure my heartbeat at 100 bpm and it's reading around 50. I see some cool looking UI stuff but the actual measurement isn't accurate.
I made my own heart rate app (using gemini at first and then Claude for lots of further edits). https://xosh.org/heart-rate/https://github.com/SMUsamaShah/heart-rate it's all offline. UI needs more work but me and my wife are the only users therefore it doesn't matter that much.
At one point I added the same EVM based heart rate detection but that requires sitting very still. I use it on my phone mainly therefore the common finger method is easiest one to use.
It's still vague beyond saying they trained an AI: "VitalLens is trained on the PROSIT and Vital Videos Africa dataset...The time-synchronized sensor array used for PROSIT consists of a video camera, electrocardiogram (ECG), pulse oximetry, blood pressure monitor, and an ambient light sensor. "
The response I anticipate will be "But this will help doctors over telehealth and stuff!" - Please see https://calebhearth.com/dont-get-distracted