Project VISUM

VISUM was our second-year IoT group project at NSBM: a kiosk that looks at your face, analyses your skin, and recommends products. You don't touch it. You stand in front of it and move your hand, and the pointer on the screen follows.

Getting that hand tracking to work was the best thing I did on it, and the part I'm proudest of.

Gesture control from a depth camera

The sensor is an Xbox 360 Kinect, which was never meant to be a Linux mouse. The first prototypes used a webcam and MediaPipe hand tracking; the kiosk itself uses the Kinect's depth camera instead.

The engine converts each depth frame to distances and keeps only a thin slice of space in front of the kiosk. Whatever is inside that slice is, in practice, a hand reaching toward the screen. It takes the centre of that shape as the pointer position, and the average depth as how far the hand is pushed in.

Raw positions from a depth camera jitter, so the pointer is smoothed before it moves the real mouse. After that the kiosk's interface needs no special support: it is an ordinary web page being pointed at.

The engine around it

There was no ready-made framework for any of this, so I wrote one. It is called the VISUM Detection and Recognition Engine.

• It keeps the Kinect alive. A monitor thread watches the USB bus and reports when the Kinect drops off or comes back. If the sensor fails to start, the engine uploads its firmware again and retries.
• It reports its own health. Each module (Kinect, Bluetooth, the Arduino, the server) writes its state, and the interface asks the server for it, so the screen can say what is wrong instead of freezing.
• It feeds the face camera. The Kinect's colour image is filtered down to skin and published as a virtual webcam, which is what the interface's face analysis reads.
• It can be driven by hand. A small command prompt turns the mouse control and the debug views on and off while it runs.

The rest of the kiosk

The server answers the interface's requests: it scores a face from its landmarks, finds the nearest Bluetooth device so a returning visitor is recognised, and produces recommendations. The recommendations come from a language model that first looks things up in a teammate's research on skin classification, so the advice is tied to that research and not made up.

An Arduino handles the physical side: an ultrasonic distance sensor and the buttons. The interface is a Next.js app with pages for the scan, the beauty score, the assistant and a visitor's history.

It was a team project with Yehan Namarathna and Tharundi Pawani Perera; Bluetooth and parts of the face analysis are theirs.