In this Psychology Report, I talked with Dr Michael Philipp of Massey University School of Psychology here in New Zealand. Michael and his students are interested in how emoji and emoticons are used in computer based communication – so, those emails tweets and Facebook posts – that kind of thing.
You can listen to the podcast as well as reading our conversation below.
I start by asking Michael how we became interested in this in the first place.
MP: My background is out of communication studies, and in communication one of the things I became really interested in was the extent to which a lot of the cues we use to behave and interact with other people are non-verbal cues. A lot of our emotional intentions but also our social intentions are communicating non-verbally and that really got me interested in the persuasion and attitude change literature around how we go about changing people’s attitudes, changing people’s behaviour on an everyday basis. As I have pursued that in communication, I realised that a lot of that work was happening in psychology and so I switched over to psychology. A lot of my work looks at how we use nonverbal expressions, whether those are facial expressions or behavioural gestures, use of space and things, to negotiate our social interactions, but the idea of emoticons is a quite interesting one in that it’s very much a nonverbal medium that happens in a verbal medium – sort of this conjunction of the idea of having symbols and texts that we’re used to typing but it’s not decoded – our brain doesn’t see it as text, we see these emoticons as faces. And, in fact they’re actually processed very differently than words are processed and they have a lot of the features of faces. So, I had this conversation with some students of mine a few years back who volunteer in our lab here and they got interested in it, because, of course, these are emoticon natives now these twenty year olds coming up. And many of them are just sort of new to the idea of emoji as well and so it’s kind of that blending of these emotional icons and they got interested in how we process these as well, and how we go about communicating emotional and social information through these texts channels – whether that’s via texts on your phone or through email and seeing how the influence people differently than words and how how much more impactful they can be.
SJ: Sure, because there’s two ends to this isn’t there? There’s the person who is writing the message or putting up some kind of posting on the web and their intention around their communication, but also the receiver as well, and how that message impacts upon them. how they process that information. You mentioned a couple of things there; the emoticon, and the emoji. So, can you maybe explain the difference between the two?
MP: Sure, yes. So, emoticons are probably – if you’re over the age of 30 – you’ve probably used or seen an emoticon at some point in your life if you’re using the internet, probably email especially, and this is where as you’re typing along in a message you might see a colon those two dots vertically aligned followed by a bracket to the right of it that can end a closed bracket which makes them a a smiley face turned 90 degrees. Now, if you’re over 35 or 40 you’re probably used to putting a dash in between that colon and bracket and if you are under 30 you might find the colon-dash-bracket very disturbing. This is something we found as well. So, there’s a culture around creating these but originally back in the … gosh, it was even before the 1970s – well it was back in the 1970s seventies these emoticons started on these local bulletin boards that people had set up when they’re talking to each other and these emerged as ways of giving the nonverbal kind of equivalent of a smile that we are normally given in communication in a very kind of emotion-less medium. And so we use that a lot now to convey the intention of a message, to help sort of relay the positive feelings behind it, or perhaps negative feelings behind it, where the words might be a little more ambiguous. Emoji on the other hand are these very graphically almost animated – I guess some of them animate and some don’t – but these orange round faces that look very much like smiley faces – they’re indistinguishable; they’re not text space, they don’t have the brackets, they’re just these usually yellow heads that are smiling, frowning, increasingly doing all sorts of other things. And so the emoji are sort of the grandchildren of the emoticon in many ways and then that’s turned into a lot of other stuff as well. So, it’s emoticons I think are probably still around, but even if you try to type an emoticon into Microsoft Word or other programs these days, it’ll turn it into an emoji pretty quick – it’ll turn it into a little icon because it thinks that that’s what you’re looking for is a smiley face and trying to make a little more compatible. So, I imagine over time these emoticons will become less popular as the emoji take over. But, I think it’s still interesting at an emotional processing level, thinking about the psychology of it – how the emoticon still have the effects they did, but also looking at as people transition into emoji what that means for the symbols they’re using and how those are interpreted, as you say, both from the sender and the receiver side of things.
SJ: It is interesting how rapidly it is changing isn’t it? I’ve seen some operating systems now – I’m thinking particularly Apple Operating System, the the mobile one, which is what I’m most familiar with – sometimes whole chunks of text are replaced by emoji, and it’s not just facial emoji, its other representations as well. So, it’s really interesting how that’s rapidly changing the nature of communication and, as you say, I don’t think we really understand how that’s perhaps impacting upon the intention and how that communication is received.
MP: Yes, absolutely. I mean there’s the intention of the receiver, potentially the sender, but of course as you mentioned, there’s also how we’re now mediated by the the platform that were using, and one the things we’re are finding with emoji is that different platforms sort of have different representations of these emoji. So, underlying there’s the universal code set up for these emoji and each platform has an icon that correlates with that code. But, of course, each platform doesn’t have to have identical ones and so recently Apple switched its gun emoji to a squirt gun – it’s much more sort of playful gun than other platforms use – that leaves much more realistic looking guns, right?
And so you can imagine that the platform used to send you a gun emoji from my iPhone and received on your Samsung, you know we could cross meanings quite easily and have miscommunications because of that. I think similarly with the face emoji, how we can communicate emotional expressions, there’s a lot of talk about what different ones of these mean despite the fact that they have labels, you know, that are underlying them. But do we all make the same meaning out of them? And do people – especially when you’re doing them cross-platform – are there opportunities for misunderstandings.
Variation in emoji
SJ: That’s really interesting because I guess what you’re saying there is that actually introduces perhaps more ambiguity into that communication where one of the theories that I think that you’re interested in is that the intention around the use of emoticons is actually to try to reduce ambiguity, to allow more effective communication.
MP: Yes and I think there is a tension, there will always will always be a tension. You can look at just the example of using a dash in the traditional emoticon. So you have a colon-dash-close bracket which looks like a smiley face on the side and the dash sort of stands for the nose and over time that sort of disappeared. And one of the things we found is that people who are used to the dash don’t interpret the colon-close bracket as a smiley face as quickly – they don’t kind of have the same kind of automatic response to it as a positive symbol as people who are used to the other way around. And so it does become inculturated – some kind of an understanding, and those understandings become shared understandings in the group that use them in a particular way. And that becomes quite different to our facial expressions which we learn quite widely, whether they’re perfectly universal or not, they’re quite broadly shared – the idea of smiles being these affiliative gestures. We put on a smile when we want to get along with other people and yet when they become these these icons, whether they’re emoji or emoticons, they have this more localized meaning in the group that we’re sharing them with. And, as you say, over time these get to be adopted in certain uses in groups and when that kind of colloquial happens in a group where that meaning is not shared more broadly a lot of miscommunication can happen.
SJ: So, there’s a couple of things that interest me in what you’re saying. I was just reminded that I’ve come across sometimes people who use an emoticon the other way around. So, rather than using the colon and then bracket, whether they have the dash in there or not, they actually use the bracket and then the colon and through further investigation just on my part, that seems to be more common with left-handers.
MP: Oh really?
SJ: They may use that configuration the other way around. So, there’s an interesting detection that I’ve noticed, because I was really confused by it – what does that mean – there are cultural differences around how perhaps that’s expressed maybe.
MP: Yeah, and that actually kind of points to another interesting thing which is this idea of how we process faces. So, we know from a lot of the face processing literature how we go about perceiving a face, and perceiving emotions in other people. We are really good at holistically examining a face – meaning we don’t look at the individual features of a face – we take it all in at once and we can make sense quite quickly of who that person is, the gender of the person, the ethnicity of the person perhaps, and also the emotional state of the person. We do that quite well all at once, not by looking at individual features of the face but by taking it all at once, and we know this because we can actually interfere with holistic processing quite well by simply doing things like a inverting face. So, if I flip a face upside down I become really bad at doing all of those things quite quickly. We just interrupted our ability of to holistically process faces. You can do it in a number of ways but inverting a face – turning upside down – is usually the easiest way. And one of things we found with the emoticons is the same thing. If you invert it – if you do exactly what you described by doing the bracket-colon, people don’t recognise it if they’re not used to it that way. They don’t recognize it as an emotional expression and the meaning is isn’t as impactful because it doesn’t moderate the tone of the message in the same way as if it’s the way we were used to it. And it’s interesting as to how people then become used to doing it that way is through others; that a small group of people doing it that way or people who are maybe new to emoticons that are trying it out for the first time and don’t realize they’re is sort of culture around which way you do it. Left-handed thing? I’ve never looked at the left-handed thing – that’d be really interesting to see! But I completely identify with feeling bewildered by looking at those the first time.
SJ: It is curious, isn’t it? Because you could end up committing all these cultural faux pas by coming in, blundering in and using these emoticons whereas actually perhaps there’s a different set that’s being used within that particular group of people or culture. And also it doesn’t work that well, in my experience, where perhaps there’s irony or sarcasm that’s being expressed in a chain of emails. Often it may actually increase the ambiguity or can result in people actually looking a little foolish in that situation.
MP: Yeah, absolutely. I think the variety of emoticons mean you can use the semi-colon to make it wink in place of the colon. But then you look at the emoji and it’s just a numbing number of faces and sentiment that you can send. You look at some of these and think, I don’t even know what this expression means, and of course then you get one like that you’re not sure what it means. And it becomes a very simple communication problem of sender and receiver, you know, what is the sender encoding – what do they mean to send – and what is that the receiver decodes don’t necessarily need to be the same thing. That becomes a really interesting communication problem, thinking about how do we coordinate meaning between the two? Is it meant to be coordinated? Is it sometimes the fact it makes us feel good to put people on edge by making them feel uncertain? There’s a lot there.
Emoji and mimicry
SJ: Yeah, there is a lot there and its really again that relationship between the sender and the receiver and it’s been played out not just in the text but also the perception of what particular punctuation marks and emoticons mean. Which kind of brings me on to what you were trying to do, I guess, in the work that you were doing was to look to see what the influence of these emoticons might have upon the receiver, but particularly by looking at some physiological reactions that, to a certain extent, are beyond conscious control. Is that right?
MP: Yes, so one of the things that we look at, and being interested in facial expressions in particular, one of things we look at is the muscles, we measure the muscles that people use to make facial expressions. So, we can use sensors called electromyography sensors – the process is called electromyography or EMG which helps us look at the micro-contractions of muscles associated with smiling or frowning and various other expressions. What we did was we essentially monitored the smiling muscles on people’s faces and as we showed them these different emoticons to see if people had these automatic mimicry reactions. Now, of course, we can all smile volitionally, we can all put on a smile quite easily but we’re not thinking about it we find in fact positive and negative situations we smile more or smile less in these muscles. And we may not be apparent looking at a person or smiling more smiles but these muscles actually kind of ‘charge up’ more in positive situations and less in negative situations, and so if we can monitor those muscles we can kind of see these micro changes in people’s propensity to smile or not smile in a given situation. What we find is essentially when we people see emoticons that they’re used to seeing in the orientation they’re used to seeing them, with the configuration are used to seeing, they do give up an automatic smile within about three to four hundred milliseconds – they reciprocate that smile very much like we do in normal situations and normal interactions. And that’s a mimicry response – we use these mimicry responses to help negotiate our interactions with other people and if you give them emoticons are not used to seeing, whether it’s including a dash when they’re not expecting it between the colon and bracket, or inverting it, or turning it the other way – doing a bracket-colon as you as you mentioned – we see that people don’t give this mimicry response. In fact, often you don’t see any response at all even after a good number of seconds because people are still trying to figure out what it is that you’ve shown them. And this is different to showing people the word ‘smile’, for example. We show the people the word ‘smile’ and we don’t see the same response. So this is very much a response that’s that’s similar to looking at other faces and that’s that’s sort of interesting.
SJ: It is interesting to me because I’m going to guess the underlying idea here is this idea of contagion of emotion – that you can actually stimulate the receiver to experience a particular emotion through your communication and not through the text of your communication but through this mimicry of facial expression that they are cued to receive – either through some kind of innate process as you mentioned, or through a learned process. They’ve learned to group these bits of punctuation together, they recognise that as a face, and then they have this reaction in their smiling muscles that is signalling this contagion, this communication of emotion and their experience of that.
MP: It’s not necessarily the fact it’s not going to make everybody feel better about things you have to say – obviously if you have a pretty dark message you know it’s not going to moderate that whole message and make people feel better about it, but I think it does give the sort of natural smiling that we do every day when you’re communicating with people, it sends that same sort of feeling. I mean, I never used emoticons until I started teaching online and you realise that you know often you just need it there because there’s a lot of times we are offering constructive feedback and you want to be taken in a positive way and and the emoticons help. Of course, the platform I use for online teaching converts it instantly to emoji but that’s ok. But I think it’s an important thing that moderates that helps to kind of regulate the meaning and help people understand that they’re on the right track, that it’s not being sent with no feeling whatsoever.
SJ: Have there been any experiments done, that you know of, where there is a clash between the meaning and the emotion that’s perhaps generated by the text and then followed by a contrasting emoticon that is quite clashing with that textual message and how they are perceived ,or how the message is moderated by that? Are you aware of anything like that?
MP: I’m trying to think of other emoticon ones and I don’t know of any offhand. There’s a lot of instances of nonverbal communication – examples of just other nonverbal communication and communication literature – that look at – your gestures and tone of voice and things as you deliver news to a person and having some clash between that. Having a very negative message but conveying a very positive tone and these kind of things, and there’s some evidence that the nonverbal kind of element of that really up-regulates, kind of makes it a more positive message and I think whether true or not, there’s there’s this notion in the communication literature that you know the nonverbal signals are in fact kind of the majority of the meaning conveyed in any piece of communication. So, it’s very difficult to talk to somebody and give positive news without any sort of positive effect or expressions – it’s not taken as positive then. And I think that’s probably important thing here: I don’t know how much they can conflict before it seems just insincere or, as you were talking about before, sort of sarcastic but I think it matters a lot. It would be be an interesting study.
SJ: Yeah, I think it does matter a lot. I mean I guess the things that are popping into my head are often people who struggle with communicating around things like change management. That there may be a top line where this is going to be beneficial for their business or the company or the organisation but actually when you work it down there are some quite serious changes, perhaps, that might be experienced. And how that message is nuanced but also perhaps followed-up, inappropriately or appropriately, with nonverbal emoji / emoticon messages through emails inadvertently or purposefully… I mean, I would be very curious to see how that is received.
MP: Yes – I can speak about what I thought was unrelated but anyway related now area of interest of mine which is looking how we make sense of other people’s expressions and one of the things to look out for in that situation that we find is that people who are kind of low in feelings of belongingness, people who might be being exploited or feel exploited, have a stigmatised identity a social group. They actually are much better and keener at distinguishing more sincere and less sincere emotion in other people. And and so one of the things you always look out for is this idea that, you know, people get kind of cynical when they feel like they might be taken advantage of, or they might be sort of a low-power situation, and in these cases I think the attempt at a positive emotion expression without any sincerity behind it can backfire in people. That’s always something that’s gotten me quite interested recently is looking at those instances where we try to put on a fake smile to comfort somebody but it actually makes things worse, because a person can tell that it’s not a genuine felt emotion and I wonder that about emoji too – there’s so many variants of these emoji – can they backfire if they come across as insincere?
SJ: I think you’re right. I think we’re really good at sniffing that stuff out and if the purpose of the emoji is to reduce ambiguity, often I think if it used contextually insensitively it can actually reduce ambiguity but in a way that counters what are you are actually intending.
MP: And it has everything to do with coordinating that meaning and I think that the important thing is, as a sender, realising that you know the meaning you’re trying to send – you’re trying to coordinate that with other people and they may actually need to see something or hear something different than what you want to say in order for them to understand what you want to do, or the meaning you want to convey. Which is always hard, right? this is why I think you can’t participate in online media these days without kind of understanding at least the basics of emoji and emoticon and what means what, because otherwise you’re out of touch and you’re not actually gonna be able to communicate with large swathes of your online community now and that can be really confronting for people who have never used these iconic representations of emotions before.
Why do emoji matter?
SJ: So, if we think about the the basic findings here in the research that you’re doing – this idea that the emoticons seem to be processed in a similar way to faces – they have this kind of configuration that people recognise quite immediately and it provokes this mimicry response on their faces, this spontaneous smile if they see another smile, if it kind of makes sense within the context of the message as well. So, who should care about something like this? What’s the point? What are the implications of a finding like this?
MP: I think it is a good question – it’s an interesting thing for the students who are studying it because they’re they’re using it all the time … on one hand, so long as it’s interesting to the students who are learning about psychological research and learning to ask these kinds of questions I don’t care what they do as long as they’re interested in finding questions that are relevant to them. On the other hand I think it gives us some food for thought in using these icons in our everyday lives. It gives us a sense that when we use them we have to be using them very deliberately and understand the community that we’re using them with. They’re increasingly becoming very localised languages of their own, but with that said, not languages – they’re being processed more fluently and more emotionally than than language perhaps can be. And so I think it’s about being deliberate with the use of them, realising that not everyone is going to be using them as fluently as we are especially coming from a different community than us. And I think so probably at the end of the day, it’s about just being very deliberate and considerate in the use of of emoji and emoticons and perhaps trying to be more understanding of faux pas that people might make in the use of them. Increasingly, there’s news stories about miscommunications with emoji, or the representation of certain emoji representing certain causes – whether that’s the Alt-Right or whether that’s the crying emoji that people are increasingly calling out as as being very inconsiderate, or schadenfreude emoji, emoji that people use to kind of laugh at others pain. And these are all really interesting because of course they weren’t designed to mean these things, but they’re being used in a particular way and not everybody realises that. And, of course, those are instances of emoji use that are probably more universal than most. Yeah, I think it’s just opening the door to a lot of questions about the use of these things in everyday life.
SJ: I think you’re right. I think for me the evolution and the change the emoticons where you have had to learn to group those things together and the more you did that the more quickly you responded to them – it seems that is what you’re saying – but that’s changed now to these emojis which are much more direct representations of faces that we recognise without having been exposed to this kind of acculturation of what a colon and a bracket looks like together, what that means. So, perhaps we’re much more quick to react in an emotional way to these emoji now then we would have been to those emoticons. So, perhaps we’re entering into this new phase and, like you say, where these emojis are deployed for purposes that are very different to what the actual facial representation appears to mean.
MP: Yeah, but I think one of the kind of universals here across emoticons and emoji is this idea that these graphics, these icons, stand out quite a bit, they stand out more than words. There’s an interesting phenomenon with language around how we can’t not read words when we see them. It’s always kind of one of these horrible things being on Twitter is getting spoilers by simply accidentally looking at a tweet, right? You can’t unsee it once you’ve seen the words, right? We automatically read words but what’s interesting is that I think these icons, these emoticons and emoji, actually stand out even more obviously and we can’t not see them when they’re in that piece of text. And, so that that smiley face at the end of your message to somebody whether it’s an emoticon or emoji, it’s almost certainly processed well before you get to the end of the message – it will stand out, it pops out to us, just like faces. We have face effects where we see faces – we readily see faces in the world around us, even if they’re not really there, right? We look at the clouds and can see a face or lots of good instances of that and I think one of the things, one of the interesting kind of uses of emojis and emoticons is that they can they can accent a message, they can they can draw out particular idea, particular emotion or sentiment quite quickly and readily as a person looks at a piece of communication. And I think there’s probably a middle ground there were too many emoji sort of ruin it – make it hard for us to get that – you know that it turns it into something else and I think there are lots of popular kind of games of know turning entire messages- you’re writing entire messages with emoticons and it takes a lot more work, I think. Whereas, kind of accenting your message with the important bits highlighted with emoji or emoticons is probably a really useful way of conveying sentiment. But, you know, this is maybe an old man speaking and it might well be that in 20 years my son will be writing everything in emoji and it will understand it perfectly! But I think at the moment at least, I’m thinking of them as accents, as emotional accents, meaningful accents, that convey ideas perhaps more emotionally, more strongly than the words themselves is a really useful way of thinking about them,
SJ: That’s a curious analogy to me – let’s let’s stick with that emotional accents for a second. Because, often if we hear an accent we can become a little bit curious or maybe even slightly obsessed with trying to figure out where that person’s accent is from and we may not actually pay too much attention to the content of what they’re speaking about, and I’m just thinking about the emojis … is there any evidence that perhaps if say it was located at the end of a tweet, we prioritise looking for and decoding that information and then we actually perhaps lose the content, or the content is then imbued with the tone of the emoji that we’ve actually processed with priority?
MP: Yeah, I can’t say anything in particular, but almost certainly assume that’s happening. I think especially in a small short piece of of communication that has a smiley at the end of it where we can look at the whole written segment and we think we see an emoticon at the end, that emoticon will pop out well ahead of time and ideally does trigger a particular frame of mind, does trigger an emotional context for the tweet, or whatever we’re reading. I have to believe it so – I haven’t done this yet but I think it’s worth looking at, for sure.