Face Replace Activity

Can Mathematica act as an alternative to Image Editing Software?

Working with Mathematica’s Image Processing functions to overlap, position, alter, and edit images.

Introduction

Initialization

tearsofjoy=;
pouting=;
surprised=;
neutral=;
smiling=;
frowning=;
image=;
image2=;
image3=;

FacialFeatures[image];(* If running FacialFeatures here takes too long or it causes an error, run the initialization cell without it *)

Goal of Activity

Replace all of the faces in a picture with emoji’s. First random emoji’s, then emoji’s that correspond to the face’s emotions.
Steps to do so: 
	Make sure emoji pictures are the same size and are transparent
		- Learn how to use Mathematica’s Image Processing functions
	Learn how to combine FindFaces and FacialFeatures with ImageCompose 
		- Learn how to use Mathematica’s FindFaces & FacialFeatures machine learning functions

ImageTake - Removing Unnecessary Parts of Images

tearsofjoy

ImageTake takes a specified amount of rows and columns from the image. The first list argument is the specified rows (height), second is the specified columns (width).
- Tip: one way to easily estimate how much to crop/take is to first look at the Image Information window for image dimensions and to hover over the preview of the original image to see pixel coordinates.

tearsofjoy2=ImageTake[tearsofjoy,{1,200},{1,200}]

pouting (* 322 x 160 img *)

Because we only want one of the pouting emoji’s not both, we want to cut the width in half, so we will ImageTake only around half of the columns.

pouting2=ImageTake[pouting,{1,160},{1,160}]

ImageCrop - Cropping Margins of Images

Margins in our images will make the resizing of the emoji’s to the size of faces in pictures inconsistent. Let’s crop them off.

pouting2

ImageCrop[pouting2]

emojis={tearsofjoy2,pouting2,surprised,neutral,smiling,frowning}

Map allows us to apply ImageCrop to every emoji in the emojis list.

emojis=Map[ImageCrop,emojis]

RemoveBackground - Creating Transparent Backgrounds for Images

Next, all we have to do is make the backgrounds of the emoji’s transparent, so we will use the Map shortcut /@ to apply RemoveBackground to all the emoji’s in the emojis list. The below code is equivalent to doing: Map[RemoveBackground, emojis]

emojis=RemoveBackground/@emojis

Now emoji pictures are all set, we need to figure out how to overlap them onto the faces of images.

FindFaces - Moving onto Face Detection

FindFaces will return Rectangles with different coordinates that represent the face locations.

FindFaces[image]

Note (Version 12.0): Running the above may give you a strange C Compiler error like the one shown below, but this doesn’t seem to produce any errors so don’t worry about it.

Show[image,Graphics[FindFaces[image]]]

Instead of rectangles, we want to replace their faces with emoji’s instead.

Working with Rectangles - Adjusting emoji’s to properly overlap the squares

boxes=FindFaces[image]

Rectangle takes in two arguments, the bottom left vertex coordinate and the top right vertex coordinate of the desired rectangle.

1) Resizing emoji dimensions to face dimensions - Finding the widths and heights of each face rectangle

Width will be the difference of x-coordinates in each list in Rectangle, height will be the difference of y-coordinates in each list.

r=Rectangle[{1,1},{4,4}]

r[[1]]

This will return the difference between the two x-coordinates in the Rectangle r, showing the width to be 3.

r[[2]][[1]]-r[[1]][[1]]

Now we will use this to find the widths and heights for all of the face squares in the image. The # you see in the below maps represents the usage of a pure function - basically in each iteration of the Map, the element will go in place of the #. The ampersand (&) just declares that a pure function is being made. Here is a simple example where each element in the list is added to one.

#+1&/@{1,2,3}

widths=#[[2]][[1]]-#[[1]][[1]]&/@boxes

heights=#[[2]][[2]]-#[[1]][[2]]&/@boxes

Huh, all of the rectangles are actually squares. We don’t actually need the heights variable then.
Note: In version 12, it seems that FindFaces is more accurate and the rectangles are not actually all squares.
Note - An alternate way to find the width would be the square root of the area of each rectangle.

ImageCompose - Overlapping emoji’s over the faces

2) Finding the locations of the faces to correspond to emoji’s

ImageCompose will overlap the image in the second argument over the image in the first.

ImageCompose[image,smiling]

Places center of emoji at {200, 200}

ImageCompose[image,smiling,{200,200}]

Hmm, FindFaces only gives us the vertices of the face squares through Rectangles, so...
We will calculate the middle of the face squares to place the center of emoji’s there

Create a list of the coordinate positions of the centers of each face square by just using RegionCentroid on boxes, the list of Rectangle coordinates.

centers=RegionCentroid/@boxes

Show[image,Graphics[FindFaces[image][[1]]]]

ImageCompose[image,emojis[[1]],centers[[1]]]

Nice! Now we just need to resize the emoji.

ImageCompose[image,ImageResize[emojis[[1]],widths[[1]]],centers[[1]]]

Instead of just the crying laughing emoji, let’s use a random emoji so that it will be different each time.

RandomChoice[emojis]

ImageCompose[image,ImageResize[RandomChoice[emojis],widths[[1]]],centers[[1]]]

Now to make an image that will finally replace all of the faces with emoji’s. Because you can only ImageCompose one image at a time, we need to do several ImageComposes on top of each other using Table.

Table[x,{x,1,5}]

How many times should we repeat the ImageCompose? The amount of faces that there are in the image. Use Length of FindFaces (boxes) to find this number.

Length[{1,2,3}]

Length[boxes]

RandomChoice[emojis]

Final Product 1 - Random Emoji’s

imageFinal=image;
Table[imageFinal=ImageCompose[imageFinal,ImageResize[RandomChoice[emojis],widths[[n]]],centers[[n]]],{n,1,Length[boxes],1}]
imageFinal

Combined Code:

(* emojis set up *)
tearsofjoy2=ImageTake[tearsofjoy,{1,200},{1,200}];
pouting2=ImageTake[pouting,{1,160},{1,160}];
emojis={tearsofjoy2,pouting2,surprised,neutral,smiling,frowning};
emojis=ImageCrop/@emojis;
emojis=RemoveBackground/@emojis;
(* findFaces set up *)
boxes=FindFaces[image]; (* can replace image *)
widths=#[[2]][[1]]-#[[1]][[1]]&/@boxes;
centers=RegionCentroid/@boxes;
imageFinal=image; (* can replace image *)
Table[imageFinal=ImageCompose[imageFinal,ImageResize[RandomChoice[emojis],widths[[n]]],centers[[n]]],{n,1,Length[boxes],1}]
ImageCollage[{image,imageFinal}]

Instead of random emoji’s ... Let’s match their facial expressions (different image)

image3

FacialFeatures (Versions 11.3 and up) may take a while to load, especially if not evaluated at the beginning in initialization. 
It may also work incorrectly and identify all the faces with the same features. Not sure what to do about this though.

FacialFeatures[image3]//Dataset

FacialFeatures will return the predicted emotions of the faces as entities.

emotions=FacialFeatures[image3,{"Emotion"}]

emotions[[1]]

emotions[[1]][[1]]

#[[1]]&/@emotions

emotionsWords=#[[1]]&/@emotions

Instead of RandomChoice[emojis] let’s create a function with Switch that takes in the emotion and returns the matching emoji.

emojis

Switch[2,
1,"a",
2,"b",
3,"c"]

InputForm[emotionsWords[[1]]]

match[emotion_]:=Switch[emotion,
Indeterminate,emojis[[4]],
Entity["Emotion", "Anger::bb6bt"],emojis[[2]],
Entity["Emotion", "Fear::7k5tk"],emojis[[3]],
Entity["Emotion", "Surprise::dj3w9"],emojis[[3]],
Entity["Emotion", "Neutral::7468h"],emojis[[4]],
Entity["Emotion", "Happiness::d8973"],RandomChoice[{emojis[[5]],emojis[[1]]}],
Entity["Emotion", "Sadness::t2566"],emojis[[6]],
Entity["Emotion", "Disgust::s4t7h"],emojis[[6]]
]

emotionsWords[[6]]

match[emotionsWords[[6]]]

Final Product 2 - Emotion-Matching Emoji’s

All that was changed was instead of RandomChoice for picking emoji’s, we used a match function that matches emotions to corresponding emoji’s.

boxes=FindFaces[image3]; (* can replace image w/ different one *)
widths=#[[2]][[1]]-#[[1]][[1]]&/@boxes;
centers=RegionCentroid/@boxes;
imageFinal=image3; (* can replace image *)
Table[imageFinal=ImageCompose[imageFinal,ImageResize[RandomChoice[emojis],widths[[n]]],centers[[n]]],{n,1,Length[boxes],1}]
ImageCollage[{image3,imageFinal}](* can replace image w/ different one *)

Combined Code

(* emojis set up *)
tearsofjoy2=ImageTake[tearsofjoy,{1,200},{1,200}];
pouting2=ImageTake[pouting,{1,160},{1,160}];
emojis={tearsofjoy2,pouting2,surprised,neutral,smiling,frowning};
emojis=ImageCrop/@emojis;
emojis=RemoveBackground/@emojis;
(* findFaces set up *)
boxes=FindFaces[image3];(* can replace image w/ different one *)
widths=#[[2]][[1]]-#[[1]][[1]]&/@boxes;
centers=RegionCentroid/@boxes;
(* facialFeatures set up *)
emotions=FacialFeatures[image3,{"Emotion"}];(* can replace image w/ different one *)
emotionsWords=#[[1]]&/@emotions;
match[emotion_]:=Switch[emotion,
Indeterminate,emojis[[4]],
Entity["Emotion", "Anger::bb6bt"],emojis[[2]],
Entity["Emotion", "Fear::7k5tk"],emojis[[3]],
Entity["Emotion", "Surprise::dj3w9"],emojis[[3]],
Entity["Emotion", "Neutral::7468h"],emojis[[4]],
Entity["Emotion", "Happiness::d8973"],RandomChoice[{emojis[[5]],emojis[[1]]}],
Entity["Emotion", "Sadness::t2566"],emojis[[6]],
Entity["Emotion", "Disgust::s4t7h"],emojis[[6]]
]
imageFinal=image3; (* can replace image *)
Table[imageFinal=ImageCompose[imageFinal,ImageResize[RandomChoice[emojis],widths[[n]]],centers[[n]]],{n,1,Length[boxes],1}]
ImageCollage[{image3,imageFinal}](* can replace image w/ different one *)

Side Concept: Making the emoji’s transparent to see the actual faces underneath - this is kinda weird looking so..

SetAlphaChannel will change the transparency of an image - however if there are already transparent parts to the image, they will be turned black, so we use RemoveBackground after to remove them again.

SetAlphaChannel[image,.5]

emojis2=SetAlphaChannel[#,.6]&/@emojis

emojis3=RemoveBackground/@emojis2

match2[emotion_]:=Switch[emotion,
Indeterminate,emojis3[[4]],
Entity["Emotion", "Anger::bb6bt"],emojis3[[2]],
Entity["Emotion", "Fear::7k5tk"],emojis3[[3]],
Entity["Emotion", "Surprise::dj3w9"],emojis3[[3]],
Entity["Emotion", "Neutral::7468h"],emojis3[[4]],
Entity["Emotion", "Happiness::d8973"],RandomChoice[{emojis3[[5]],emojis3[[1]]}],
Entity["Emotion", "Sadness::t2566"],emojis3[[6]],
Entity["Emotion", "Disgust::s4t7h"],emojis3[[6]]
]

Combined Code

(* emoji set up *)
tearsofjoy2=ImageTake[tearsofjoy,{1,200},{1,200}];
pouting2=ImageTake[pouting,{1,160},{1,160}];
emojis={tearsofjoy2,pouting2,surprised,neutral,smiling,frowning};
emojis=ImageCrop/@emojis;
emojis=RemoveBackground/@emojis;
(* findFaces set up *)
boxes=FindFaces[image];(* can replace image w/ different one *)
widths=#[[2]][[1]]-#[[1]][[1]]&/@boxes;
centers=RegionCentroid/@boxes;
(* facialFeatures set up *)
emotions=FacialFeatures[image,{"Emotion"}];(* can replace image w/ different one *)
emotionsWords=#[[1]]&/@emotions;
emojis2=SetAlphaChannel[#,.6]&/@emojis
emojis3=RemoveBackground/@emojis2
match2[emotion_]:=Switch[emotion,
Indeterminate,emojis3[[4]],
Entity["Emotion", "Anger::bb6bt"],emojis3[[2]],
Entity["Emotion", "Fear::7k5tk"],emojis3[[3]],
Entity["Emotion", "Surprise::dj3w9"],emojis3[[3]],
Entity["Emotion", "Neutral::7468h"],emojis3[[4]],
Entity["Emotion", "Happiness::d8973"],RandomChoice[{emojis3[[5]],emojis3[[1]]}],
Entity["Emotion", "Sadness::t2566"],emojis3[[6]],
Entity["Emotion", "Disgust::s4t7h"],emojis3[[6]]
]
imageFinal=image; (* can replace image *)
Table[imageFinal=ImageCompose[imageFinal,ImageResize[RandomChoice[emojis],widths[[n]]],centers[[n]]],{n,1,Length[boxes],1}]
imageFinal

Side Concept: Face Replace Yourself!

Let’s try using our program on ourselves! 
CurrentImage[] will take a picture through your webcam.

me=CurrentImage[]

(* emoji set up *)
tearsofjoy2=ImageTake[tearsofjoy,{1,200},{1,200}];
pouting2=ImageTake[pouting,{1,160},{1,160}];
emojis={tearsofjoy2,pouting2,surprised,neutral,smiling,frowning};
emojis=ImageCrop/@emojis;
emojis=RemoveBackground/@emojis;
(* findFaces set up *)
boxes=FindFaces[me];(* can replace image w/ different one *)
widths=#[[2]][[1]]-#[[1]][[1]]&/@boxes;
centers=RegionCentroid/@boxes;
(* facialFeatures set up *)
emotions=FacialFeatures[me,{"Emotion"}];(* can replace image w/ different one *)
emotionsWords=#[[1]]&/@emotions;
match[emotion_]:=Switch[emotion,
Indeterminate,emojis[[4]],
Entity["Emotion", "Anger::bb6bt"],emojis[[2]],
Entity["Emotion", "Fear::7k5tk"],emojis[[3]],
Entity["Emotion", "Surprise::dj3w9"],emojis[[3]],
Entity["Emotion", "Neutral::7468h"],emojis[[4]],
Entity["Emotion", "Happiness::d8973"],RandomChoice[{emojis[[5]],emojis[[1]]}],
Entity["Emotion", "Sadness::t2566"],emojis[[6]],
Entity["Emotion", "Disgust::s4t7h"],emojis[[6]]
]
imageFinal=me; (* can replace image *)
Table[imageFinal=ImageCompose[imageFinal,ImageResize[RandomChoice[emojis],widths[[n]]],centers[[n]]],{n,1,Length[boxes],1}]
ImageCollage[{me,imageFinal}]

Hmm... Let’s see if we can get different emojis by changing our facial expressions into different emotions. 
ImageCapture[] is another webcam picture function  that helps with capturing several different pictures by providing a preview through the webcam.

ImageCapture[]

Another idea: let’s try to label the emoji with what emotion was recognized. 
If we are to ImageCompose text onto an image, we need to get Text, turn it into Graphics, then turn it into an image. 
This unfortunately ruins some of the quality but it is the only way as of now.

Image@Graphics@Text@Style["anger",12,Bold,FontFamily->"Comic Sans MS"]

Right now, the background of the text image is all white and we want it to be transparent, so let’s use RemoveBackground again.
This time, we will add the argument, {“Background”, White} at the end, which specifies what color in the image we want to turn transparent. This will help us get rid of the white inside and in between the  letters.

text=RemoveBackground[Image@Graphics@Text@Style["anger",12,Bold,FontFamily->"Comic Sans MS"],{"Background",White}]

Now let’s image compose this on top of an emoji.

ImageCompose[emojis[[4]],text]

If we want to position the text somewhere else, we can specify the coordinates in ImageCompose

ImageCompose[emojis[[4]],text,{73,12}]

Now that we have this down, let’s map the correct emotion to the face, creating a text function to change what text will show and changing the match function from before.

text2[txt_]:=RemoveBackground[Image@Graphics@Text@Style[txt,12,Bold,FontFamily->"Comic Sans MS"],{"Background",White}]

match4[emotion_]:=ImageCompose[
Switch[emotion,
Indeterminate,emojis[[4]],
Entity["Emotion", "Anger::bb6bt"],emojis[[2]],
Entity["Emotion", "Fear::7k5tk"],emojis[[3]],
Entity["Emotion", "Surprise::dj3w9"],emojis[[3]],
Entity["Emotion", "Neutral::7468h"],emojis[[4]],
Entity["Emotion", "Happiness::d8973"],RandomChoice[{emojis[[5]],emojis[[1]]}],
Entity["Emotion", "Sadness::t2566"],emojis[[6]],
Entity["Emotion", "Disgust::s4t7h"],emojis[[6]]
]
,text2[emotion]
,{73,12}
]

match4[Indeterminate]

match[Entity["Emotion", "Neutral::7468h"]]

Finally, we can combine this with the finished code from earlier.

Combined Code

me=CurrentImage[]

(* emoji set up *)
tearsofjoy2=ImageTake[tearsofjoy,{1,200},{1,200}];
pouting2=ImageTake[pouting,{1,160},{1,160}];
emojis={tearsofjoy2,pouting2,surprised,neutral,smiling,frowning};
emojis=ImageCrop/@emojis;
emojis=RemoveBackground/@emojis;
(* findFaces set up *)
boxes=FindFaces[me];(* can replace image w/ different one *)
widths=#[[2]][[1]]-#[[1]][[1]]&/@boxes;
centers=RegionCentroid/@boxes;
(* facialFeatures set up *)
emotions=FacialFeatures[me,{"Emotion"}]; (* can replace image w/ different one *)
emotionsWords=#[[1]]&/@emotions;
text2[txt_]:=RemoveBackground[Image@Graphics@Text@Style[txt,12,Bold,FontFamily->"Comic Sans MS"],{"Background",White}] (* text *)
match4[emotion_]:=ImageCompose[
Switch[emotion,
Indeterminate,emojis[[4]],
Entity["Emotion", "Anger::bb6bt"],emojis[[2]],
Entity["Emotion", "Fear::7k5tk"],emojis[[3]],
Entity["Emotion", "Surprise::dj3w9"],emojis[[3]],
Entity["Emotion", "Neutral::7468h"],emojis[[4]],
Entity["Emotion", "Happiness::d8973"],RandomChoice[{emojis[[5]],emojis[[1]]}],
Entity["Emotion", "Sadness::t2566"],emojis[[6]],
Entity["Emotion", "Disgust::s4t7h"],emojis[[6]]
]
,text2[emotion]
,{73,12}
]
imageFinal=me; (* can replace image *)
Table[imageFinal=ImageCompose[imageFinal,ImageResize[RandomChoice[emojis],widths[[n]]],centers[[n]]],{n,1,Length[boxes],1}]
ImageCollage[{me,imageFinal}](* can replace image w/ different one *)

Adory Vo
adoryvo.github.io | www.abbymath.com
CT@TP: Computational Thinking at Torrey Pines High School
March 2019 - Last Updated: May 2019, Version 12.0