Here's a 2 min video that explains the project in a more exciting way so you don't need to read everything below.

Like, Comment and Subscribe!!

Abstract

Special education children often have trouble speaking in long coherent sentences. However, speech therapy is not widely accessible to everyone because it’s so expensive. Thus, Speech Supporter is a Natural Language Processing Program that teaches children how to speak in proper English.

First, the program uses Google Cloud’s Speech API to convert the user’s speech into text. Then it parses through each word to determine its part of speech to suggest synonyms to form an intelligible sentence. Lastly, to determine relevance, the program accesses Semantic Databases to perform Vector Analysis between two sentences. For instance, if the user is prompted by “What food do you want to eat?“ and the user responds with “Me like monkeys", the computer will respond with a correction. The program analyzes the part of speech of “Me” and the relationship between “food” and “monkey” to prompt the user to try again.

Inspiration

Two of our members' close family members have low functioning autism, which case them to speak rarely or speak in a non-sequitur incoherent sentences. Specifically, when asking them questions, they will usually respond with only one or two word responses that do not match the context of the conversation. Other times they were an attempt at answer the question that was impeded with a lack of language skill.

Because these individuals with special needs are unable to communicate their needs, they rely on their caretakers to translate their thoughts.Because special needs children have such expensive therapy that is often not supported by government funds, special needs families are financially burdened to send their children to speech therapy on top of the other private therapy and classes.

In today's world where soft skills are more necessary than ever, we wanted to use modern natural language processing (NLP) techniques to help children with language processing issues. To make speech therapy widely accessible and affordable, we wanted to teach children to communicate in full coherent sentences and gain confidence in themselves in their language skills so they require less reliance on their caretakers.

How It Works

The program comes with a set of open-ended questions that are designed to let the user talk about themselves, as well as example answers. When the program is running, the computer asks a question to the user, and the user should speak into the computer.

The computer then uses NLP-based methods to check for proper grammar, as well as relevancy to the question. If the user says something that is irrelevant or non-sequitur, the computer will prompt the user to answer again in an attempt to correct the user. For instance, if the user is prompted by "How are you today?" and the user responds with "I like monkeys", the computer will respond with a correction. However, if the user says something that is relevant but does not have proper grammar, the computer will attempt to create an improved sentence that uses the example answers as a template and substitutes in the new words, which the computer will them prompt the user to repeat. When the user repeats the proper sentence, the computer will indicate a success, and moves on to the next question.

How We Built It

We used Python as the foundation for this project. For both text-to-speech and speech recognition, we used Google Cloud's libraries, which integrated well with Python. To grammar check, we used linkgrammar, an implementation of the Sleator/Temperley/Lafferty theory of natural language parsing of link grammar, where a sentence is constructed through relations (links) between different words.

To check relevancy of the answer to the question, we used NLTK's Word2Vec as well as Sent2Vec through vector analysis which indicates the relevance from a scale from -1 to 1. If the user does not meet the relevance threshold, it will ask the user to try again.

Finally, in order to do intelligent substitution, we used the Spacy Module to detect parts of speech to substitute and synonyms to suggest a intelligible sentence. Both LinkGrammar and Sent2Vec use sophisticated machine learning methods to detect coherence in sentences and similarity between sentences respectively.

What We Learned

This project required a deep understanding of linguistics that we often overlook. We learned computational linguistic tactics such as how to tokenize, stem, and lematize words to manipulate them properly.

This project prompted us to think about our thinking process when it comes to speaking in order for us to replicate an accurate AI. The thought in which individuals with special needs must go through to create coherent sentences is something we often take for granted.

We learned to develop a user-centric product that is tailored to their specific needs, we learned to put ourselves in their position to adapt to our audience.To identify areas our users might struggle in, we drafted flow charts to individualize each step of our thought process. Subsequently, we broke up our words into decision trees to formulate proper syntax and diction.