We have built a chatting app for people with all sort of disabilities, such as hearing problems, blind people and elderly.
We made a chat based on touch, no headphones and no readings necessary.
How did we do it?
We used the MYO arm band in order to send and receive messages.
MYO can recognize gestures made by the hand and in our app,
you can signal in Morse code with your hand and our algorithm will interpret them to letters and eventually to text messages.
This new system allows people who can’t see, can’t hear or speak write messages and stay connected with our modern life. All they need is to signal with their hand the letters.
What about receiving messages?
In order to help people with hearing and sight disabilities we wanted to create a system of receiving messages without using sight or hearing, so,
we transformed our modern phone into a Morse transmitter.
Our app translate hand movements into text, send the text message, receive and broadcast it using
the vibration of your phone so u don’t have to wear headphones or see the phone.
You can get massages while your phone is in your pocket!
We want to make a substitute chatting app so that people with disabilities can connect with the rest of the world.
Technology is all about connecting people,
until now it meant connecting people from different physical places,
we want to generalize this goal to connect people who communicate differently.
Let’s Dive In
Our client side consists of seven java classes.
First of all, we have the HelloWorldActivity which is the main java class that communicate with the XML (GUI) and the rest of the classes. This class is the main class of this project.
Next is the Code class which is only an enum of gestures that we built in the app like: DOT, LINE, END, BLANK (such that one can write in more with this signs alone).
The Algorithm part: we have built four java classes for the algorithm.
The first one is the node of our search trie tree: TrieNode.
This class represent a node in the trie tree and is consists of boolean variable indicates if the node is a leaf, char value which indicates the value of the node (actual character)
and a map from Code (signs generated by the gestures) to another node in the trie tree.
The second class is the Trie class. This class has one private member which is the root of the trie tree. Trie is responsible for searching in the tree as well as inserting.
The TextToMorse class responsible for translating incoming text to morse in order to transmit the message using the phone vibration system.
Its members are the letter which we want to translate and an ArrayList of Code (enum) in order to create a sequence of signs. The class has a getMorseCode function which returns this sequence.
Finally, we have the Parser class responsible for the trie, its initialization. We had an idea to switch between Morse and Braille depending on the user preference and here is a bit of code from that initial idea. Based on the input value the trie is initialized on signs from Morse code or Braille (translated to dot and spaces).
class maind is for checks only and is there in case some one will want to use our code and would want an example of a morse message.
We used a known frame work for the java server called: OCSF. We are recommending this framework as it is very simple and good way to build a server.
The AbstractServer class maintains a thread that waits for connection attempts from clients. When a connection attempt occurs it creates a new ConnectionToClient instance which runs as a thread. When a client is thus connected to the server, the two programs can then exchange Object instances.
An instance of this class is created by the server when a client connects. It accepts messages coming from the client and is responsible for sending data to the client since the socket is private to this class. The AbstractServer contains a set of instances of this class and is responsible for adding and deleting them.
This class acts as a subclass of AbstractServer and is also an Observable class. This means that when a message is received, all observers are notified.
The ObservableOriginatorServer is a subclass of ObservableServer that sends OriginatorMessage instances to its observers. This class should be used when the observers need to know the originator of the messages received. The originator is null when the message sent concerns the server.
A message class used by the Observable layer in order to conserve information about the originator of a message.
AdaptableServer is an adapter class that extends the AbstractServer class in place of the AbstractObservableServer.
This class overrides some of the methods in the abstract superclass in order to give more functionality to the server.
Services this thread’s client by repeatedly requesting a
screen name until a unique one has been submitted, then
acknowledges the name and registers the output stream for
the client in a global set, then repeatedly gets inputs and broadcasts them.
How does it look
One of our main goals is to make the GUI feel intuitive** so that people who will use the app will have an easy time adjusting to it.
In order to achieve this goal, we make the GUI as simple as possible.
First of all, we have the title – the name of the app – Cactus. Then we have the built in pairing algorithm that connects the MYO to the phone using Bluetooth.
Below that we have on the left side the gesture bar for testing and training this app. This text shows the user live what gestures are recognized by the MYO in order to help writing more fluently.
On the middle we have the lock mechanism – allows the user to lock the app so the MYO gestures won’t affect.
Below this bar is the main area. All the area in the middle of the screen is used for receiving messages in an easy way.
In order to hear messages simply tap this area, anywhere you want. This GUI really makes things a lot easier for people who can’t see for example.
Lastly the text above the text area – is the received message. This only exists for helping and training users, in the end, we want users to understand the message using only the vibrations of their phone.
The text area displays the text that is been written. This helps people who do have vision and can review their message.
The paper airplane on the left is the send button, again only for people who can see, people who can’t have a special gesture to send the message (holding a fist for three seconds, see manual** below).
Connect to the MYO armband
Start writing using your hand
fist – dot
finger spread – line
wave in – delete letter
wave out – complete letter
holding a fist for three seconds – send message
You can also write normal texts in the text box below and
hit send with the button on the left
To view a message please tap the middle of your screen
the message will be transmitted to your phone via vibration
Link To Our Project Files