Skype Ubiquity bot

A bot listening to a skype converstation being able to identify and extract information from multiple sources when they are mentioned. Analyzing content of emails and images in Gmail and Dropbox accounts.


     Right from the start of this project I was really productive and I am going to run you through my first hour. In that hour I have read 3 relevant articles, which is pretty good. But I have managed to achieve so much more than that. I found the simplest and most famous recommendation letter, it was for a graduate program at Princeton and all it said was : “He is a mathematical genius.”. I have also had a look at the ranking of the best UK companies to work for according to Glassdoor and found a position I might apply to. The second place of this ranking was taken by ARM where an old friend of mine works now. Haven’t spoken to him in a while I have decided to message him to see how he was doing, achieving even the social aspect.
     So overall in just that one hour I have managed to : Learn one unless fact; Find a job and not apply to it; Send 57 Facebook messages; And read 3 articles whose content I forgot. With the sense of productivity, achievement and a bit of tiredness I rewarded myself with lunch. Therefore my question was : Can we actually multi-task? And why do we feel the urge to do everything and be everywhere at once?


     Unsurprisingly research shows that humans cannot multi-task. Even the term multi-tasking was first created for computers in the 70s and adopted for humans. A more fitting term would be task switching which is what happens in our brains. When presented with multiple sources of information our brains quickly switch focus from one to another. This isn’t really good for us, when switching between multiple tasks such as watching a tv whilst working on an assignment our IQ can be decreased by almost 10 points [1], having the same effect as being sleep deprived or a little high. Due to this, studies have shown a clear correlation between task switching and lower academic performance [2]. What can be even more worrying are the possible long-term effects lowering one’s EQ [3]. In the short-term effects, up to 40% of time is wasted during multi-tasking [4].
     Pre frontal cortex is a part of our brain responsible for deep focus, so what happens when we try to multi-task? First we need to make a lot of small decisions : “Should I check my Facebook?” and “Which message should I reply to first?”. Since the pre frontal cortex has only a limited about of memory it can keep at a given time it needs to readjust to the new contest, expending time, energy and releasing stress hormones which can damage it over time [5]. But if multi-tasking doesn’t work, why do we feel such a sense of achievement at the end?
     Ironically the part of our brain responsible for deep focus is also really prone to distractions and loves novelty [6]. The fact that the devices we use in the process have been designed to maximize this dopamine release reward does not help, effectively creating an addiction-loop.
     There are a lot of articles on how to increase productivity : “Get more sleep”, “Meditate” and “Get rid of distractions competing for your attention”. Looking at my very, very cluttered desktop I could really see the case for the latter, so I got rid of all the open programs but the ones I really needed to work. Removing all the distractions like email, videos and Facebook has definitely increased my productivity. Problem solved! … Well not really.
     I found that a lot of resources and information I need to work is spread across different platforms, spaces and times. With all of my will-power mustered up I stopped myself from visiting these time-wasteful services. But I still managed to get lost, one second I was reading a relevant article on Linkedin, the other I was endorsing a person I barely know for a skill they probably don’t have. And the same happened with Facebook, I had to go and ask a friend for help, but whilst waiting for their reply I started a conversation with 3 other people. This is the problem. Even with increased will power some things cannot be prevented. We lose time by having to focus on navigating through a different world to retrieve a piece of information relevant to our current context. Even further, whenever we venture into these worlds and realities we are risking getting lost in them due to their well-engineered addictiveness. This doesn’t happen only when one is working. I was having a coffee with a friend when I mentioned a photo I have taken. So I unlocked my phone and searched for it, navigating a different reality without being able to focus on the conversation I was having. After showing the photo, with the phone in hand I habitually checked my Gmail and messenger, involuntarily sucked in again to the different realities.
     Why do we have to travel between worlds, spaces and times? Why can’t they just come to us?


     One solution would have been getting rid of technology and being more determined, but I believe that technology can be both a problem and a solution. Could there be a ubiquitous entity standing between our reality and all the other ones? Pulling information from these different worlds into ours when we need it, saving us from having to venture ourselves. Turns out there is a solution like this, artificial intelligence bots. One can just say “Ok Google” or “Hey Siri” and they can carry a task for us, not quite. All of these still require one to venture into their world as they can speak only when spoken to. I would have wanted a bot that understands when it is needed and then carries out the task without me having to interacting with it. In the coffee scenario I would have wanted to take my phone and as I unlocked it the photo would be there immediately, saving me from having to venture into a different world and risking getting lost. And that is what I decided to make.

     The Skype Ubiquity bot enables two or more people to have a conversation through Skype as they normally would, mimicking a scenario of sitting in a cafe, with the only difference that the bot now listens to what is being said. Once it hears a prompt it can help with such as “I have seen the cutest panda photo today” it gets to work. Transcribes the sentence into text, understands the core intent – open photos, extracts time – today and extracts the content – panda. Then it goes searching through one’s Dropbox folder, looking at pictures from today which have a panda in them and if it finds one it sends it in chat. The same idea goes for emails where in can analyze and understand content of each email in a mailbox and respond to a sentence like “Do you remember the email I sent you yesterday about the sports article?”. Lastly it can also respond to agreeing the next time when the group wants to meet up by opening a calendar.
     As a conclusion, the Ubiquity bot stands between you and all the other worlds created by technology. Once it hears a sentence it can help with, it ventures into these worlds for you, retrieving the relevant information. Saving you from carrying out this task yourself and risking getting lost.

Technological background

     This bot is an adaptation of a project I have worked on with two other friends. It links multiple individual tools and joins them in the context of this bot. More specifically powered by the Microsoft bot framework as a platform to integrate to Skype, listen and send messages. Using Microsoft TextToSpeech to transcribe the text to speech. LUIS as a language model for understanding the intents and entities in a sentence or a paragraph. Cognitive services to be able to analyze text and images in order to extract and match them to the entities from LUIS. Integrated with means to interact with Dropbox and Gmail. This is all joined in one server running Node with python scripts.

[1] Rubinstein, J., Meyer, D. E., & Evans, J. E. (n.d.). Executive Control of Cognitive Processes in Task Switching. PsycEXTRA Dataset. doi:10.1037/e537272012-189
[2] Junco, R., & Cotten, S. R. (2012). No A 4 U: The relationship between multitasking and academic performance. Computers & Education,      59(2), 505-514. doi:10.1016/j.compedu.2011.12.023
[3] Horsnell, M. (2005). Why texting harms your IQ. The Timesonline.
[4] Meyer, D. E. & Kieras, D. E. (1997b). A computational theory of executive cognitive processes and multiple-task performance: Part 2. Accounts of psychological refractory-period phenomena. Psychological Review, 104, 749-791.
[5] Levitin, D. J. (2014). The organized mind: Thinking straight in the age of information overload. New York, NY: Dutton.
[6] Sanbonmatsu D, Strayer D, Medeiros-Ward N, Watson J. Who multi-tasks and why? Multi-tasking ability, perceived multi-tasking ability, impulsivity,      and sensation seeking. PLOS One. 8(1), e54402 (2013).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s