UE4 Emote Shader (Facial Expression Flipbook) *Updated!*

Tutorial / 15 December 2018

For the past year, I have been working on a VR game as a student project. The premise of this game is that you are a student in wizard school trying to gather information and ingredients for a test. Due to it being in VR and tech limitations we were trying to think of smart ways to have the students convey emotions and facial expressions without doing a full face rig. At the time Animal Crossing came out on mobile. Then it dawned on us to have all expressions be UV animated and textures.

So this began my journey.


1st - Reference & Research


I gathered as much reference as I could of games that did this to make sure it would even look good in a 3D world. Zelda games and a lot of other Nintendo games through the years have always used a method like this so I was not worried. Then I came across a game in development currently called Billie Bust Up that has a video of how they implemented their 2D eyes.  (link: https://www.youtube.com/watch?v=pvLG4rmkB8A&feature=youtu.be ).


The only catch was our game could not have that many joints in the faces without causing troubles since our original goal was to have 20 students on screen at any given time. BUT this gave me the idea that I could do this and there was hope.



2nd - Prototype


I worked for a while trying to get things to work in the UE4 shader network - then I stumbled upon flipbooks. Flipbooks are a way to project sprite sheets onto a 3D object - so it would be a way for us to animate faces or any other object. So I figured out a rough way to implement it and did a rough prototype to show to my team. The designers were blown away and we moved forward from there thinking about what expressions we should use, how the students should act, etc. I talked to the character artist on our team and they began to make the sprite sheets while I began to figure out the shader.



3rd - Shader Functionality


The emotes are controlled by a named Scaler Parameter, mine is named Emote. That way the designers/programmers can use it in the AI. The parameter has a min and max value and a default value. I plug this into every if statement into A.


For the If statements the B is whatever value I want them to be. I have comments in my shader so I remember which value each is and so do my programmers.

I use a Lerp to plug in my skin color and my emotes using the alpha channel. My skin color is in a material function due to us having 3 different skin tones and randomizing students. Whatever you want to be shown as black in alpha channel plugs into A, whatever you want to show up as white plugs into B.

For your first emote I plugged my Lerp into both A==B and A<B due to the fact that I want this to be my only option, I also have a constant value for A>B for all emotes.

For the next emotes I do the same thing except I plug the last emote's if statement into the next if statement in A < B so they are always connected.

Note: The emote should always plug into A==B due to whatever value your if statement is set to. When the scaler parameter changes to that value it will show that emote. (So in my case 1 = 1 will show default face)


Now for the flipbooks, I used a material function just to help with space.

The Flipbook is very simple. I just needed to tell it the number of rows, columns, and speed my animation was going to be. Then plug the flipbook node into the UV's. After that make sure for the material function had another output for alpha and name it correctly. You can play with the speed a little bit and get some interesting things.

By the end, I had something like this.





4th - Result


In the end, I had something like this. The shader had 5 emotes, 2 of which were animated through flipbook. We also used this technique to put emotes on a hat in the game and on a few other things.



5th - Performance & Draw call

Due to needing to lower the draw call I ended up returning to the emotes and making them cheaper. I put each animated emote on their own flipbook and all single image emotes on a full sprite sheet. I then did the math so the Emote Parameter could skip between the still images for that flipbook then go to the animated flipbooks with no problem.

Here is the material blueprint so it may make a little more sense.

Here is also the final result!