While I was working on my own version of a ubiquitous computer and enabling devices to seamlessly interact with one another, I found out about the Web of Things and one of the main labs working on it: Simon Mayer's Interactions Lab.

That place looked like a dream to me back then: a whole lab whose goal was improving human-to-machine and machine-to-machine interaction. It was the perfect place to work on my mandatory semester project, the perfect continuation of the work I had been doing with Omnia, and the perfect starting point for what I was planning would be Omnia's next chapter [link to come].

All the rooms in the lab were furnished with sensors and actuators connected to the Internet. The ubiquitous computing dream of "computers that disappear into the fabric of everyday life" had already become reality there. All this data was being collected, and these devices were easily accessible — but nobody was doing much with it. That's how my new project, "Learning from interactions", came to be.

A hand-drawn diagram of the Learning from Interactions system: SENSORS feed data into a MODEL (a neural network) that produces a prediction; an APP turns the prediction into an output, which is compared against the user's input to produce an error that is fed back to the model.

The idea

The idea was that people shouldn't have to keep repeating the same actions to modify the environment according to their preferences: turn off the light, turn on the A/C, lower the blinds, switch on the humidifier, and so on. The environment should be able to observe the user, learn their preferences, and adjust itself accordingly. This is what "computers embedded into the fabric of everyday life" really meant to me — and finally, with enough sensors and a bit of machine learning, this dream could be realized.

The badge

I started acquiring data from a room in the lab, but I needed something to help me distinguish the person in the room so I could associate their preferences correctly. Inspired by the DSTIKE watch I used for Omnia, I created a little badge that people in the lab would wear to be identified as they moved around and to record their preferred work environment.

Acquiring data from a room in the lab: VS Code running the code that reads the IoT sensors and works out which device a user is near, alongside the InfluxDB / Chronograf explorer showing a year of sensor readings. The little badge — an Omnia-branded board on a lanyard — that people in the lab wore so the system could identify them and record their preferred work environment.

Training the system

The setup was really simple. During training, whenever a user performed an action — like turning on a light — they would also press a button on their badge to tell the system that they had made that choice, so it could learn who liked what.

A slide titled "Acquire personal preferences" showing the training loop: (1) the light is switched and (2) the badge button is pressed, both reported to the Manager, which (3) gets the current sensor data as a snapshot and (4) uploads it to the database.

Within a few days — enough time to collect the training data — the system was correctly identifying the preferences of the various people in each room and adjusting their environment accordingly, without any intervention on their part.

A terminal log of the system learning: repeated "Wrong prediction :(" entries, then it retrains a new model on 944 datapoints, after which the prediction turns to "Correct prediction :)".

The best technology disappears

It was a simple demonstration of a concept I deeply believed in: the best technology is the one that disappears. After this project, I wanted to automate everything!