Rhythmic interface with machines

Introduction

Many forms of human user interfaces are available to interact with electronic devices. Most commons ones use a keyboard and mouse and touch screens. Voice interface is developing and gaining momentum.

This project was started from the observation that most commonly used HID require that the user has a visual contact with the interface device (applies in particular with touch devices like smartphones and tablets, also with keyboards).

Some common or recurring tasks however could be completed without a visual contact with the device if the interface was designed for that. These task include launching a specific app, opening a file, writing a standard text...

The project will explore an ancient and yet rich, not to say endless means of communication which are rhythms.
Drums have been used to carry messages between villages. Some people living in rural areas use song and whistle as a second specialist second language to communicate on a longer distance than voice would allow.

Rhythm can be performed by various parts of the human body and can be recorded through an even larger range of means.

There is a lot of fun trying to find out how this could be used to interact with existing electronic devices, computers, tablets, machines and change the user's experience...

Developing the project

As starting point, a prototype code and hardware environment have been developed. The intent is to get a functional interface device at hand to explore the potential uses for it. Also, the second objective is to share the project with other enthusiasts to develop it to further stages.

The system was modeled as follows:

A - the actual interface on which the user performs a rhythm
B - convert inputs into rhythmic patterns
C - lookup for the translation of the rhythm pattern in a dictionary
D - interpret translation and perform the action

First implementation

All 4 basic tasks are performed by a java code running on a computer. Input is pthrorformed using 5 keys of the computer's keyboard dedicated to rhythmic input.

See project article

Second implementation

The interface device becomes independent from the device that will perform the action.

An arduino uno has been used as interface device, using switches and / or microphones as input detectors.

The communication with the computer is done through USB : arduino mocks up a USB keyboard.

The computer runs exactly the same code as for first implementation
*** not yet finalised ***

see project article

Third implementation

The interface device is similar to the one used for the second implementation, but instead of mocking a keyboard, the device communicates in background with the host device with a serial communication, over bluetooth.

The programme on the host device is tailored based on the one used previously. The main difference lies in the fact that the programme listens to serial input instead of keyboard input.

The expected benefits are to be able to run this programme as a background task.

under development

see project article

To be developed : integration of more steps in the interface device

Integrate one more step in the interface device : convert input into rhythm and send phrases to the host through serial. Expected benefits are enhanced reliability since delays in serial communication are a drawback of the third implementation.
needs further development of the Arduino code
see project article

Integrate the next step in the interface device : translate the rhythm - can open usages of the interface device in full authonomy in a connected objetc, without a host computer or device.

To be developed : link with a programme like AutoHotKey to perform wider range of actions on the host computer

AutoHotKey is an open source programme running on Windows that allows to automate actions and trigger them through keyboard shortcuts.

It could act as a straight forward companion to the interface device, replacing the keyboard shortcuts by a rhythmic input.

To be developed : smartphones

Create a rhythmic listener on a smartphone, using screen or sound or accelerometer sensor, and allow to perform some recurring functions.

Comments

Processing - playing with particles collections

Colourful ant-like particles description Particles created, move, interact, grow from encounters and disappear. click and drag to create new particles Processing code: Automated vehicles reaching target and avoiding obstacles description Processing code: Coil spring description Processing code: Spring 2D description Processing code:

Implementation 3 - use arduino uno and bluetooth link

concept: Function A (input) performed by an arduino uno connected through bluetooth as using a serial communication. Other function B C D actions performed by a java programme. Image of hardware setup: Arduino with switches, bluetooth adapter, serial link The main benefits of this configuration is that the keyboard is not needed anymore as a link in the input chain. The host can then listen to the bluetooth data sent in the background, while the keyboard is being used for different purpose. Also, the application on the host does not need anymore to have the focus. Last, the input device can be split from the host computer or device.

Processing - drawing interface good for kids

Drawing interface using processing 3 Description This sketch allows to draw shapes simply. Click once to activate the pen. An initial color is selected. Press an arrow button to select color and mix cyan, magenta, yellow, black using arrows. Press SPACE to get back to drawing. Draw as long as the pen is activated moving the mouse. Then click again to deactivate the pen. Each line can be erased by pressing SPACE. Erasing takes place in the reverse order of it being drawn. code Processing:

Arduino projects, sensors and actuators

Moving average on analog input Description This small project shows how to implement a moving average with a lightweight form in order to smooth raw analog values. The value is then used to position a servo Image code Arduino: Using Ultrasonic sensor Description In this project, we are using a US sensor. Very first step to understand the behaviour of the sensor through the serial interface. Image code Arduino:

Home Automation - Garage Door Security

Description: This is the research project to understand the home automation system which I would like to develop when I'm building my own house in the future. Burglary is a common threat while living in South Auckland. My wife's mountain bike was stolen from the garage in our first year living in the South Auckland. Even though we have moved to different house and living in better neighbourhood now, always worried and paranoid about the garage door. So this project developed to monitor the garage door and alert if the door is open. Some of the terminology used in this post: Home Assistant - Home Assistant is an open-source home automation platform running on Python 3. Track and control all devices at home and automate control. Many components are supported well by Home assistant community (Wemo, Amazon Echo, Google Home, Google Assistant, Zigbee, MQTT, IFTT, etc..) MQTT (aka MQ Telemetry Transport) is a machine-to-machine or “Internet of Things” conn...

Drive your lego model by voice

Voice controlled Lego merry-go-round Build a model using lego and a motor Drive the motor at various speed using an arduino board as variable frequency drive Send commands to the arduino board using your voice, through a Processing sketch on your computer and bluetooth That's it, and it works Description hardware A Lego motor - old motor from Lego technics - DC current - 4.5V Arduino - Uno in our case Bluetooth board - ref Transistor Diode Breadboard and jumper cables Arduino code : Processing code :

Implementation 1 - java based rhythmic listener and interpreter

All 4 steps performed by a java programme Java was selected since it allows one programme to run independently of the platform, Windows, OSX, Linux. A The user can perform a rhythm using up to 5dedicated keys on the keyboard. B, J, K, L, M were selected since they fall easily enough under the fingers of a right handed user. Entries are performed in a way close to playing keys and chords on a piano keyboard. B keystrokes on the dedicated keys are converted into "chords" and "phrases" "Chord" represents the keys that are pressed simultaneously - it is basically an array of 5, with values 1 or 0. "Phrase" represents a group of chords separated by a variable interval. This is the actual rhythmic pattern. The length of the interval between chords is recorded using a number. The shortest interval is given the value 1, the other intervals are given a multiple (2 if the interval is twice as long as the shortest in the phrase, ...) Th...

ABC game - letters sounds and rhymes

Audio interactive game for kids around letters and numbers learning Description Toddlers are attracted to computers, they want to play with it, press the keyboard all the time when their parents use it - pretty much for everything. I was browsing the web for games designed to help kids learn their letters and ABC when I decided to take the opportunity to build a simple game allowing the child to discover the letters, while taking an active role in building the game it self. Here is the concept: The game displays a letter and at the same time plays an audio relating to the letter. If the player presses the right key at the first attempt, a congratulation sound is played and lots of progress is made. If the right letter / character is found after several attempts, less progress is made, no congrats sound is played. The child or the parents can record every sound matching the letters as well as the congrats messages. I got my daughter to record a short voice memo to ma...

Smart temperature sensor and client interface

Wifi enabled temperature and humidity sensor and client sending email notifications description This project was developed as part of a research to build a temperature automation system for our charming but cold Auckland home. Relying on electric oil heaters as sole source of heating in children's room, the aim was to monitor continuously the temperature at night and to turn on or off the heater according to the measured temperature. The project described in this post does the first half and mode demanding part of the job which is log a temperature, broadcast it to the web using the MQTT protocol through the house's wifi. A small and cheap ESP8266 Wifi board programmes using the Arduino language reads temperature and humidity through an DHT22 sensor and broadcasts the values to an MQTT server at set intervals. A custom client written in the java based Processing 3 and running on a computer located anywhere (Macbook in this case) displays the readings and sends not...

Manukau tech projects

Search This Blog