The idea
A few years ago, I saw a lovely YouTube video that I simply have not been able to find since (please help). It was by this calm fellow narrating about a simple motion detection method that he likes to use and how it uncovered interesting things about the world. He gives a very nice demo of pointing a camera across something like a meadow or maybe just down a hill where there's a meadow and some trees and some bushes.
On the camera recording itself you can't see that there's anything special going on. And then he describes this motion detection method which is to take two sequential frames, turns them to grayscale then subtract the gray color value at each pixel between frames, and where the difference is sufficient, mark the pixels as having motion and then have the video just display those pixels, or overlay them over the original.
The video thus reveals a deer in the meadow, walking around slowly, that is just absolutely not visible without this method and this stuck with me. It's been bouncing around my head for years because I've always wanted to use it as a dance tool (performative and/or pedagogical).
The how
As I wrote previously, I've been trying to increase the ambition with which I approach ideas with AI tools. Cursor is my tool of choice right now, and I managed to get a prototype working in an hour or so. I continue to be amazed at how quickly I can iterate with AI.
The toy
Motion is the toy I built, and am still building (It currently only works on Macs, or rather, WebGPU isn't fully supported across OSs and Browsers, so in many combinations it doesn't work, meaning I actually have no idea which devices it works on).
I'm so incredibly happy with it, and it's so much fun to build. This was an early aha moment:
The more I made, the more choices I needed to make, like what color the background and effect should be:
I quickly made settings for these and all other "dials" that were sprinkled across the code. For instance, how different are the two grays allowed to be to still say there's "motion" (sensitivity), or how long do the trails last (decay rate)?
Next, what if colors change over time, and you can define the speed they change? (Flashing lights warning from here on).
I don't know why it took me this long to think of it, but decay can also be 0, so that colors never go away.
I was visiting my family and this proved to be wonderful entertainment for kids. I consider it an achievement if a kid decided this is worth an hour of their time.
I also added a record button, so you can record what's on the screen and download it when you're done. (It's buggy, works about 50% of the time)
Coming up
Most recently, inspired by some Twitter/X accounts like @poetengineer__, I've started adding Google's Mediapipe models to frame processing. This lets me isolate body parts, like face, hand, and even finger:
Realtime video processing is like magic.
If you have any fun ideas, reach out to me on X or Bluesky, I'm happy to hear them.