Cool to see this, it's a cool in-between step for not having additional wraparound screens or a VR headset.

I used to run a similar software[1] for when I was really into playing F1 racing games. However one of the problems I found was the initial disconnect in your head and eye movement that took some getting used to.

For example, if you want to look left to see an upcoming turn, naturally your eyes move before your head, and your head follows after. With this software enabled, you have to consciously inverse the process where your head moves a direction, but your eyes still remain looking forward at the screen.

It took a some getting used to and resulting in some dizziness afterwards, but was fun.

[1]: https://facetracknoir.sourceforge.net/home/default.htm

just wondering, did this difference in looking around mess you up at all when driving in real life

I'd wonder if the trained reflex of turning your head to shift the sim view bleeds into real driving — not consciously, just as muscle memory after enough hours.

Not the person you're asking, but I can answer from my own experience: No. There's a big difference between looking at a monitor and looking out a real car window. Your brain can tell.

Wouldn't it be better to use head tracking to get the position of the head relative to the monitor, so the monitor behaves like a window? Like in Johnny Lee's classic Wii demo [1].

The way it currently works (rotating the view upon head rotation) doesn't really make sense because a monitor is not a head mounted display.

1: https://youtube.com/watch?v=Jd3-eiid-Uw

If it behaved like a window you wouldn't be able to look beside or behind yourself in game, this type of setup is used a lot for sim games.

But if you look behind yourself your head is pointing away from the screen.

My kid is using a webcam based head tracker with a combat flight sim of some sort. You don't want to move your head too far since you are looking at the monitor right? It works kind of like mouse acceleration where if you move your head quickly, it changes perspective further.

Yes, this tech has long been used to great effect for flight sims. It might seem odd if you've never used it, but it turns out it's very intuitive.

I loved that demo, but the problem with "monitor as a window into the world" is that monitors are relatively small and people don't sit very close to them. The FOV you obtain with most setups is disappointingly small. You need to be relatively close to a large display for it to work well. It's one of the reasons why the idea never took off in the first place, I think.

> The way it currently works doesn't really make sense

And yet, sim players are using it. Players want to use small headmovements to simulate large head movements ingame. It seems to work.

Wouldn't it be much more convenient then to buy a VR headset which actually follows your head movements because it's physically head mounted? Not to mention that it provides a stereoscopic view of the scenery. I guess the price is a hurdle.

There's actually some perception research on this. The two techniques have different goals: "window" mode preserves geometric accuracy; "look-around" mode extends the virtual field of regard. For sim racing, the latter is clearly what you want.

Also check out the SmoothTrack mobile app. Same use case but the compute is done on a phone instead of the gaming machine. Head position data can be sent over local network or USB.

Hey! Thanks for recommending this!! That’s my app. :) pushing the tracking work off to the phone is helpful for freeing up resources on the computer.

Just wanted to say thanks! The first time played Flight Simulator 2020 with SmoothTrack I was blown away. I still am blown away each time too. Great experience that’s been nearly flawless for me and really turned up the realism. I can move my head closer to gauges and see what they’re saying, look around stuff, get a better view out the window. Wonderful.

Is there specialized face/head tracking hardware on phones like there is on DSLR cameras?

Thanks for making a fantastic tool!

Thanks for recommending. I can only place my real camera in somewhat awkward spots, so this would give me freedom to adjust its position more freely.

I gave OpenPOV a try with FS 2024, and found it really disorienting. It was not useful at all. I went to a Meta Quest 3 and that actually made me feel like I was inside an aircraft. At on point I tried to lean on a bulkhead. Oops.

They're called airplanes because they're made out of air in VR!

What are the differences between OpenTrack and this ?

It was discovered and completely reimplemented independently without knowledge that Opentrack exists? That's the only thing I can figure. Except they actually mention TrackIR as that's the input method they are using.

one is generated slop and the other is a mature community project

when your head move, and with it your eyes, you move what's in front of them to your perspective inside the simulation, just as VR works

but in this case it is detrimental because the screen is fixed, the natural behavior would be not to move it

or at least do very little with it like a parallax

the current demo would cause nausea after a moment

This is a good example of having sound logic but not understanding the actual use case. It's simply a way to add functionality in a way to attempt to mimic what humans are capable of in a game. Not everyone wants to or is capable of using VR for various reasons. This allows you to use a slight physical movement of your head to replace using a mouse to move the camera, primarily in flight and racing simulators. That means you don't have to take your hand off of the racing wheel to move a mouse around, or even need to have a mouse available to you.

Is there a reason eye tracking can't work or doesn't work?

People playing simulators such as DCS are used to have head tracking with OpenTrack. It's very helpful

Have you tried it or is that your theory?

Don't all headtrackers work like this? Also the infrared ones.

When I used a head tracker (homemade infrared one), I just got used to shifting my head but keeping my eyes on the screen. Having a wider screen helps.

I learned how to shift my head only a little bit to move the FOV a lot when using one of those infrared trackers. Still kinda hurt my neck.

Tons of people use head tracking like this via TrackIR and similar setups, it's quite common for space or air sim games.

And even with milsim FPS, as those blog posts from more than 10 years ago shows: https://dslyecxi.com/category/trackir/

This is underrated advice. Wish someone told me this 5 years ago.

It looks like this is just for Microsoft's Windows. I used to use TrackIR when I gamed on that. I missed that functionality when I moved to game on Linux. That is, until LookPilot (https://lookpilot.app/, it's on Steam, too) arrived. Webcam tracking is good because you don't need to wear a headset, but not so good in a dark room.

While we're on the subject, there's also TrackyMouse: https://trackymouse.js.org/

You can do some related tricks to this in browsers too: https://github.com/atomirex/fishtank-js (example https://atomirex.github.io/fishtank-js/dist/index.html )

There is a greatly cleaner version of the same idea in the React Fiber libraries these days as well.

But doesn't it make you look away from the monitor? Unless you have this 45 inch curved monitor really close to your face?

It makes you turn your head away from the monitor – your eyes are still pointed at it. You generally configure it to apply some gain on your head rotation as well, so a 15 degree head swivel turns into a 60 degree camera movement or whatever.

Maybe think of it less like a pretend-VR solution and more like a camera controller wired to your neck muscles.

It sounds weird, it feels weird at first, but it gets quite natural after a brief adjustment period. It's also very cheap to try if you're curious.

It requires robotics monitor arm

Are you serious? That's not the cheapest setup then. Would be better to buy a VR goggles in such case

Actually, VR headsets are typically more expensive than a motorized monitor arm, not cheaper. Though I concede the whole setup does seem like overkill compared to just... getting VR.

They are not serious

It would be cool to use something like this or openfov to control OBS to automatically switch between different cameras/scenes when you turn your head. Either multiple cameras, or switching between screenshare/camera if you look directly into the camera.

Hmm, from the title i expected a tool to calibrate FOV angle to monitor size/distance

It would be nice to know the limits of this tech, like how does it tolerate head gears and garments like headphones or hoodies, beanies and glasses, long hard, different skin colour and facial features or even background contrast.

Wonder if I could use that to swipe through virtual desktops?

Would be cool to see this become a fully viable TrackIR alternative.

...why would webcam head tracking need a new standalone vibecoded project? I thought there are plenty of those already.

FaceAPI, OpenTrack, AITrack -- these have existed for years. The question's fair. What does this one do differently?

Why not? Such as?

Sometimes it's just fun to make something, avoiding it just because said thing already exists in some form is silly.

I mean, completely ignoring prior researches can be a bit disingenuous.

Why? I'm supposed to avoid projects just because someone else did a similar thing unless I want to delve into their research or something?

I can do whatever you want with my own projects, if I want to make an app to do X thing I can.

Building something for fun is fine. But "it just has to work on race day" is a support ticket waiting to happen when the camera can't find your face at midnight.