Rhythmic interface with machines

Introduction

Many forms of human user interfaces are available to interact with electronic devices. Most commons ones use a keyboard and mouse and touch screens. Voice interface is developing and gaining momentum.

This project was started from the observation that most commonly used HID require that the user has a visual contact with the interface device (applies in particular with touch devices like smartphones and tablets, also with keyboards).

Some common or recurring tasks however could be completed without a visual contact with the device if the interface was designed for that. These task include launching a specific app, opening a file, writing a standard text...

The project will explore an ancient and yet rich, not to say endless means of communication which are rhythms.
Drums have been used to carry messages between villages. Some people living in rural areas use song and whistle as a second specialist second language to communicate on a longer distance than voice would allow.

Rhythm can be performed by various parts of the human body and can be recorded through an even larger range of means.

There is a lot of fun trying to find out how this could be used to interact with existing electronic devices, computers, tablets, machines and change the user's experience...

Developing the project

As starting point, a prototype code and hardware environment have been developed. The intent is to get a functional interface device at hand to explore the potential uses for it. Also, the second objective is to share the project with other enthusiasts to develop it to further stages.

The system was modeled as follows:

A - the actual interface on which the user performs a rhythm
B - convert inputs into rhythmic patterns
C - lookup for the translation of the rhythm pattern in a dictionary
D - interpret translation and perform the action

First implementation

All 4 basic tasks are performed by a java code running on a computer. Input is pthrorformed using 5 keys of the computer's keyboard dedicated to rhythmic input.

See project article

Second implementation

The interface device becomes independent from the device that will perform the action.

An arduino uno has been used as interface device, using switches and / or microphones as input detectors.

The communication with the computer is done through USB : arduino mocks up a USB keyboard.

The computer runs exactly the same code as for first implementation
*** not yet finalised ***

see project article

Third implementation

The interface device is similar to the one used for the second implementation, but instead of mocking a keyboard, the device communicates in background with the host device with a serial communication, over bluetooth.

The programme on the host device is tailored based on the one used previously. The main difference lies in the fact that the programme listens to serial input instead of keyboard input.

The expected benefits are to be able to run this programme as a background task.

under development

see project article

To be developed : integration of more steps in the interface device

Integrate one more step in the interface device : convert input into rhythm and send phrases to the host through serial. Expected benefits are enhanced reliability since delays in serial communication are a drawback of the third implementation.
needs further development of the Arduino code
see project article

Integrate the next step in the interface device : translate the rhythm - can open usages of the interface device in full authonomy in a connected objetc, without a host computer or device.

To be developed : link with a programme like AutoHotKey to perform wider range of actions on the host computer

AutoHotKey is an open source programme running on Windows that allows to automate actions and trigger them through keyboard shortcuts.

It could act as a straight forward companion to the interface device, replacing the keyboard shortcuts by a rhythmic input.

To be developed : smartphones

Create a rhythmic listener on a smartphone, using screen or sound or accelerometer sensor, and allow to perform some recurring functions.

Comments

Processing - playing with particles collections

Colourful ant-like particles description Particles created, move, interact, grow from encounters and disappear. click and drag to create new particles Processing code: Automated vehicles reaching target and avoiding obstacles description Processing code: Coil spring description Processing code: Spring 2D description Processing code:

Processing - drawing interface good for kids

Drawing interface using processing 3 Description This sketch allows to draw shapes simply. Click once to activate the pen. An initial color is selected. Press an arrow button to select color and mix cyan, magenta, yellow, black using arrows. Press SPACE to get back to drawing. Draw as long as the pen is activated moving the mouse. Then click again to deactivate the pen. Each line can be erased by pressing SPACE. Erasing takes place in the reverse order of it being drawn. code Processing:

Home Automation - Garage Door Security

Description: This is the research project to understand the home automation system which I would like to develop when I'm building my own house in the future. Burglary is a common threat while living in South Auckland. My wife's mountain bike was stolen from the garage in our first year living in the South Auckland. Even though we have moved to different house and living in better neighbourhood now, always worried and paranoid about the garage door. So this project developed to monitor the garage door and alert if the door is open. Some of the terminology used in this post: Home Assistant - Home Assistant is an open-source home automation platform running on Python 3. Track and control all devices at home and automate control. Many components are supported well by Home assistant community (Wemo, Amazon Echo, Google Home, Google Assistant, Zigbee, MQTT, IFTT, etc..) MQTT (aka MQ Telemetry Transport) is a machine-to-machine or “Internet of Things” connectivi

ABC game - letters sounds and rhymes

Audio interactive game for kids around letters and numbers learning Description Toddlers are attracted to computers, they want to play with it, press the keyboard all the time when their parents use it - pretty much for everything. I was browsing the web for games designed to help kids learn their letters and ABC when I decided to take the opportunity to build a simple game allowing the child to discover the letters, while taking an active role in building the game it self. Here is the concept: The game displays a letter and at the same time plays an audio relating to the letter. If the player presses the right key at the first attempt, a congratulation sound is played and lots of progress is made. If the right letter / character is found after several attempts, less progress is made, no congrats sound is played. The child or the parents can record every sound matching the letters as well as the congrats messages. I got my daughter to record a short voice memo to match

Arduino projects, sensors and actuators

Moving average on analog input Description This small project shows how to implement a moving average with a lightweight form in order to smooth raw analog values. The value is then used to position a servo Image code Arduino: Using Ultrasonic sensor Description In this project, we are using a US sensor. Very first step to understand the behaviour of the sensor through the serial interface. Image code Arduino:

Implementation 2 - use arduino uno acting as keyboard

concept: Function A (input) performed by an arduino uno connected through USB and acting as a keyboard. Other function B C D actions performed by the same java programme as in first implementation. Image of hardware setup: Arduino wwith switches, USB link, Story In this implementation, we try to separate the input device from the device finally being controlled and performing the actions. Arduino uno is fitted with 5 inputs. To start with these inputs are switches. These switches are treated exactly as the keyboard keys in the first implementation. the switch pressed is considered as the 'keydown' event. Arduino sends the USB code representing a combination of keys pressed ( max 6 keys can be pressed simultaneously together). On the host side, the program still listens to key pressed and builds the "words" according to the rhythm at which the switches are pressed. Arduino code Specific trick to have Arduino Uno seen as a keyboard by a co

Implementation 3 - use arduino uno and bluetooth link

concept: Function A (input) performed by an arduino uno connected through bluetooth as using a serial communication. Other function B C D actions performed by a java programme. Image of hardware setup: Arduino with switches, bluetooth adapter, serial link The main benefits of this configuration is that the keyboard is not needed anymore as a link in the input chain. The host can then listen to the bluetooth data sent in the background, while the keyboard is being used for different purpose. Also, the application on the host does not need anymore to have the focus. Last, the input device can be split from the host computer or device.

Smart temperature sensor and client interface

Wifi enabled temperature and humidity sensor and client sending email notifications description This project was developed as part of a research to build a temperature automation system for our charming but cold Auckland home. Relying on electric oil heaters as sole source of heating in children's room, the aim was to monitor continuously the temperature at night and to turn on or off the heater according to the measured temperature. The project described in this post does the first half and mode demanding part of the job which is log a temperature, broadcast it to the web using the MQTT protocol through the house's wifi. A small and cheap ESP8266 Wifi board programmes using the Arduino language reads temperature and humidity through an DHT22 sensor and broadcasts the values to an MQTT server at set intervals. A custom client written in the java based Processing 3 and running on a computer located anywhere (Macbook in this case) displays the readings and sends not

MQTT and stuff

Journey through the internet of things world Wanted to explore and understand the ecosystem of internet of things. Started with some hardware ESP8266 ESP-01S is used in a range of relatively cheap board which can connect to a wifi network, handle some IOs and be programmed using the arduino language and software. Mated with a temperature sensor and a power regulator (supports 3.3v only), you get a small circuit that can read data and publish it. libraries used : dht mqtt wifi Where to ? MQTT is a protocol used to handle object to object communications. Light and supported by free brokers and libraries for several languages, it takes a few hours to get started and publish the first message. broker used : hivemq also mosquitto and other In our basic example, the object sends data - publishes and we retrieve the data published on a phone or website. The broker is a middle ground where the data is made available. Using the data many free IOS apps out there that pr

Drive your lego model by voice

Voice controlled Lego merry-go-round Build a model using lego and a motor Drive the motor at various speed using an arduino board as variable frequency drive Send commands to the arduino board using your voice, through a Processing sketch on your computer and bluetooth That's it, and it works Description hardware A Lego motor - old motor from Lego technics - DC current - 4.5V Arduino - Uno in our case Bluetooth board - ref Transistor Diode Breadboard and jumper cables Arduino code : Processing code :

Manukau tech projects

Search This Blog