Reframed context of Enquiry

How can we give greater tangibility to our encounters with machine learning and build more embodied relationships with such systems that capture personality within data?

My contribution:

• Conceptualized the idea of replicating personalities from Facebook messages
• Created underlying Machine Learning models
• Created NFC enabled Android application communicating with the cloud machine learning models

Exploring our Personal Data

Our personal data is rich with hauntological potential. Unlike books which are often reread and catalogued in A-Z, our personal messages are often rarely re-read as they gather and require scrolling to recover. They are arranged with preference towards what has been recently written.

Exploring such personal data can invoke a mix of emotions ranging from nostalgia to cringe and shame.

Machine Learning in practice by companies

Machine learning is already being put to use by a range of companies without the users realising their data is used to help improve them. Facebook uses machine learning to recognize faces on images. Amazon makes recommendations for your next purchases based on people who resemble you from your data.

 

Figure : Facebook image recognition and amazon recommendations

In cases where algorithms are working in such a way that doesn’t do what was intended we can explore whether it is the algorithm that is at fault or that it might be working perfectly and simply reflecting what was unintended within the input data.

An interesting aspect of machine learning is when it arrives to a point when observers are not able to tell, whether it achieved it’s purpose really right or really wrong. This is well demonstrated by the Microsoft Tay bot [Figure 3]  where it quickly turned racist and misogynist based on people’s input. At this point a question arises whether the Tay bot captured the data presented really well or whether it should have known to avoid these topics.

Additionally within Dmitry Paranyushkin’s ‘Interviews with Artificial Artificial Intelligence’ [Figure 3], the artist explores the failures of Amazon’s image recognition systems to identify a series of images uploaded, exploring the often humorous results of the system carrying out it’s sophisticated function

Figure 3 : Micorost Tay Bot and Amazon image recognition

Moving Between Data Metaphors

Figure 4. Metaphors of data in industry

The exists an interesting dichotomy between perceived values in data. Firms seeing and speaking of data as an impersonal industrial commodity. This is in contrast to how data could be perceived as something embodied (Figure 4) [1]. Metaphors move from data as oil towards something more personal, such as data as a shadow, a footprint, a body.

Design Opportunity

This introduces the opportunity to push against the ubiquity of machine learning interactions by tethering our encounters with them to a place, giving them a tactility and locality. It extends an invitation to interact with with personal data which introduces people to these more embodied metaphors for ubiquitous personal information.

Sequence to Sequence Neural Networks

For this part of the project where we were trying to reproduce personality from Facebook chat messages we used sequence to sequence words based neural networks. In contrast to the previous example the networks work based on words rather than characters, but the key part is the sequence to sequence part. Chat context is usually framed in a question and answer format. So our previous model would not work, when asked a question, we would not want our network to continue writing the question but rather try to understand it and answer it. Sequence to sequence networks are essentially two different networks one processing the questions, the other focusing on the answer. This should ensure the understanding of the question and focus on the answer. Below you can see the process of training the network (Figure 5).

 Figure 5. Conversation prototype processed through a sequence to sequence NN.

Figure 6. Distorted images of Facebook photos through Convolutional Neural Networks.

Design Methods

We intentionally removed human features from visual interface on the flags and replaced with hallucinatory images that were generated by inputting Facebook profile photos into a convolution neural network that reduced, rather than accentuated features of the input image to anonymise the appearance and humanity of the data’s author. Light-weight fabric was chosen for easy physical tagging and environmental durability, as the aim was to localize and offer accessibility.

We aimed to keep intelligence within a mobile device, instead of designing a new artifact, in order to further to afford interaction to any passerbys. In this way, we let the intellectual owner of the data listen into the conversations of their neural networks are having, also affording transparency for said data owner. The piece simulates us leaving data shadows of ourselves that have captured some part of our digital identities and placing them out into the world, manifesting ourselves as a new kind of tangible artefact.

Practise:

Figure 7 : Technical practice of Identity fabrics

The interactions starts by the user touching a flag with an NFC tag with their phone. This initiates a communication with a remote server, currently in Dublin (Figure 7). Then with a native chat interface you can interact with a remote personality.

Under the hood numbers:

Contrasting the two applications used here, Co-Authored Books used a laptop for processing, however the second project used a GPU optimized machine 200x more powerful than the used laptop. To represent a personality we would require 4x the computational power and memory. The same goes for the space used representing a personality, currently being 12Gb.

The greatest problem with achieving this convincing personality representation was the lack of message data: there was only 30,000 message exchanges. However to capture one’s personality more convincingly, around 100 000 000 message exchanges would be required [2].

Future Steps

We came up with a clever way to remedy the lack of data problem. We could use a pre-trained neural network with a general personality, for example trained on open-subtitles data. This general personality then can be trained to capture an individual. This would be done by training this pre-trained network on personal Facebook data.

Additionally, the potential for machine to machine conversations through the identity fabrics opens up questions of what these entities might talk about. Would they get into a repetition or reveal interesting progression and project personality features (extraversion, consciousness, extraversion).

References

[1] Watson, S. M., Boyle, L., Chase, S., Roso, M., Scholl, N., Toro, D.  “Metaphors of Big Data”  (Accessed on May 1, 2017) http://dismagazine.com/discussion/73298/sara-m-watson-metaphors-of-big-data/

[2] [Vinyals, O. & Le, Q. 2015. ‘A Neural Conversational Model’ Google. [online] Accessed at: https://arxiv.org/pdf/1506.05869.pdf ].