{"id":76,"date":"2018-04-26T18:29:21","date_gmt":"2018-04-26T22:29:21","guid":{"rendered":"http:\/\/mattrobertson.ca\/blog\/?p=76"},"modified":"2021-04-08T10:54:24","modified_gmt":"2021-04-08T14:54:24","slug":"turns-out-faces-are-hard","status":"publish","type":"post","link":"https:\/\/mattrobertson.ca\/blog\/2018\/04\/26\/turns-out-faces-are-hard\/","title":{"rendered":"Turns Out, Faces are Hard"},"content":{"rendered":"<p>Throughout my previous work in graphics, I haven&#8217;t really done too much related to animation: sliding spheres\/models around sure, but not animating a single rig. To amend that I decided I wanted to implement some kind of key frame-based tech for something simple so I could at least say I had done it. Having previously heard about how some of the Legend of Zelda games do <a href=\"https:\/\/medium.com\/@gordonnl\/links-expressions-eb7beae2c62c\">their <\/a> <a href=\"https:\/\/youtu.be\/hWxAw_mVASM\">eyes <\/a>, and being absolutely in love with how Wind Waker accomplishes many of its effects, it seemed like a perfect jumping off point. To start, here&#8217;s our final shader:<\/p>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.shadertoy.com\/embed\/4dKcRG?gui=true&amp;t=10&amp;paused=true&amp;muted=false\" width=\"640\" height=\"360\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>The basic idea of the whole system is pretty simple: Draw the full sclera and iris\/pupil, mask it, draw the eyelids over top, then draw in the eyebrows. The following handful of screenshots and code snippets show this process as it proceeded.<\/p>\n<p><strong>Full Sclera:<\/strong><br \/>\nAs you may notice going foward, a common theme here is to try and avoid code duplication by drawing the right side as a mirror of the left. Since all of our aspects are flat shaded, we only need to determine if something is in a specific layer (in this case, in the slera). If it is, we set its colour to the appropriate one. Due to the sequential nature of the drawing, only the topmost layer ends up &#8220;sticking.&#8221;<\/p>\n<p>The sclera itself is rendered by intersecting two quadratic equations.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-cpp\">\nbool paintSclera( in vec2 uv )\n{\n    vec2 tempUV = uv;\n    if (tempUV.x &gt; 0.5) tempUV.x -= rEyeOffset;\n    float toSquare = tempUV.x * 1.8 - .42;\n    if ((tempUV.y &gt;= .9 * toSquare*toSquare + .04) &amp;&amp;\n        (tempUV.y &lt;= -1.0* toSquare*toSquare + .34))\n    {\n        return true;   \n    }\n\n \treturn false;  \n}<\/code><\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-81\" src=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoPupil-300x169.png\" alt=\"\" width=\"300\" height=\"169\" srcset=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoPupil-300x169.png 300w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoPupil-768x432.png 768w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoPupil-1024x576.png 1024w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoPupil.png 1919w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p><strong>Pupil<\/strong>:<br \/>\nPretty simply, this is done by rendering an oval, using some reasonably basic math I Googled around to find. Note the use of pupilDilation here, when it came time to animate I wanted to be able to use a scalar factor to increase\/decrease the size of the pupil. Perhaps I should have had it as an argument instead of a global value, but I figured it didn&#8217;t hurt to have both eyes dilate\/constrict the same amount.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-cpp\">\nbool paintPupil (in vec2 uv, in vec2 pupilCenter, in vec2 eyeSize)\n{\n\tvec2 dist = uv - pupilCenter - focusPoint.xy;\n\tvec2 part = vec2(dist.x\/eyeSize.x \/ pupilDilation, dist.y\/eyeSize.y \/ pupilDilation); \/\/ (x\/a, y\/b)\n\tfloat equation = part.x * part.x + part.y * part.y;\n\tif(equation &lt; 1.0){ \n\t\treturn true;\n    }\n    \n    return false;\n}<\/code><\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-82\" src=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoMask-300x169.png\" alt=\"\" width=\"300\" height=\"169\" srcset=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoMask-300x169.png 300w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoMask-768x432.png 768w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoMask-1024x576.png 1024w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoMask.png 1919w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p><strong>Restricting<\/strong>:<br \/>\nThe reverse of the slcera layer, at this point we paint the union of the &#8220;outside&#8221; of two quadratic functions. The function itself is almost the same as the sclera, but in reverse. Later on, it gets updated to accept key-frames, so it can tween two key-frames.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-80\" src=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoLid-300x169.png\" alt=\"\" width=\"300\" height=\"169\" srcset=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoLid-300x169.png 300w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoLid-768x432.png 768w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoLid-1024x576.png 1024w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoLid.png 1919w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p><strong>EyeLid<\/strong>:<br \/>\nHere we get into a slightly more complex bit. I render eyebrows and the eyelid in the same way: as a quadratic <a href=\"https:\/\/en.wikipedia.org\/wiki\/B%C3%A9zier_curve\">B\u00e9zier curve<\/a>. As I already knew about how they worked and didn&#8217;t want to reimplement the tech, I took the implementation from <a href=\"https:\/\/www.shadertoy.com\/view\/MtS3Dy\">here<\/a>. If you&#8217;d like to learn more about them, you can check out either of those links.<\/p>\n<p>As a consequence, there&#8217;s nothing too interesting going on in my code for this: I pretty much just return true if that B\u00e9zier drawing function returns an appropriate distance from the curve.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-79\" src=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoBrow-300x169.png\" alt=\"\" width=\"300\" height=\"169\" srcset=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoBrow-300x169.png 300w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoBrow-768x432.png 768w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoBrow-1024x576.png 1024w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/NoBrow.png 1919w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p><strong>EyeBrow<\/strong>:<\/p>\n<p>Again, this is done similarly to the eyelid, but this time around I draw two B\u00e9zier curves to achieve the tapering effect I was going for.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-78\" src=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye-300x169.png\" alt=\"\" width=\"300\" height=\"169\" srcset=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye-300x169.png 300w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye-768x432.png 768w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye-1024x576.png 1024w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye.png 1919w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p>At this point, I had a decent way to render a fully open eye (my first key-frame) and needed to build a system to animate the eyes blinking. We accomplish this by creating a number of different key-frames and by interpolating smoothly between them. Luckily, when I was building the eye originally, I did it using the intersection of two quadratic functions to restrictthe sclera, while a B\u00e9zier curve paints the eyelids, this means that a key-frame ID just needs to map to the 3 points which define a B\u00e9zier curve and the 4 points to define these two quadratics, a few different key-frames follow.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-90\" src=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye2-300x169.png\" alt=\"\" width=\"300\" height=\"169\" srcset=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye2-300x169.png 300w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye2-768x432.png 768w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye2-1024x576.png 1024w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye2.png 1919w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><br \/>\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-91\" src=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye3-300x169.png\" alt=\"\" width=\"300\" height=\"169\" srcset=\"https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye3-300x169.png 300w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye3-768x432.png 768w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye3-1024x576.png 1024w, https:\/\/mattrobertson.ca\/blog\/wp-content\/uploads\/2018\/04\/FullEye3.png 1919w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p>Perhaps even more importantly, since a keyframe is defined entirely by these handfuls of floats, we can interpolate between keyframes just by lerping between the floats themselves. To do this we just need to indicate what keyframe we&#8217;re starting from, which we&#8217;re going to, and how far along that progress we are (a float between 0 and 1). You can see that, as well as how I map a shape to the important values (spoilers: poorly) below. A much cleaner solution would have been an array which used the shape ID as a key, and contained a struct containing all the important bits of info. Oh well, it&#8217;s something to consider for next time.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-cpp\">\nbool restrictPupil (in vec2 uv, in int shape1, in int shape2, in float tween){\n    vec2 tempUV = uv;\n    if (tempUV.x &gt; 0.5) tempUV.x -= rEyeOffset;\n    float toSquare = tempUV.x * 1.8 - .42;\n    vec3 A,B,C,D;\n    if (shape1 == 0){\n        A.y = 0.9;\n        B.y = 0.04;\n        C.y = -1.0;\n        D.y = 0.34;\n    } else if (shape1 == 1){\n       ...\n    }\n    \n    if (shape2 == 0){\n        A.z = 0.9;\n        B.z = 0.04;\n        C.z = -1.0;\n        D.z = 0.34;\n    } else if (shape2 == 1){\n        ...\n    }\n    \n    A.x = mix(A.y, A.z, tween);\n    B.x = mix(B.y, B.z, tween);\n    C.x = mix(C.y, C.z, tween);\n    D.x = mix(D.y, D.z, tween);\n    \n    if ((tempUV.y &lt;= A.x * toSquare*toSquare + B.x) ||\n            (tempUV.y &gt;= C.x * toSquare*toSquare + D.x))\n    {\n    \treturn true;   \n    }\n    return false;\n}<\/code><\/pre>\n<p>Lastly, I needed to actually build tech to have the eye actually blink, and move around and this can be seen below.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-cpp\">\n \/\/ This doesn't *really* focus the eyes, but it makes them slide around together\n \/\/ which I guess is close enough\n focusPoint = vec3(sin(iTime * 1.4) * .08, cos(iTime * 0.6) * .07, 0.);\n pupilDilation = abs(sin(iTime * 0.2 + 1.4)) * 0.5 + 0.7; \n\n \/\/ Blink every 4 seconds\n if (int(floor(iTime)) % 4 == 0){\n    rEyeTween = fract(iTime);\n    lEyeTween = fract(iTime);\n    if (rEyeTween &lt;= blinkDuration){\n        blinkStageDuration *= blinkDuration;\n        while (rEyeTween &gt; blinkStageDuration){\n            rEyeTween -= blinkStageDuration;\n        }\n        rEyeTween \/= blinkStageDuration;\n        if (fract(iTime) &lt;= blinkStageDuration)\n        {\n            lEyeShapes.x = 0.;\n            lEyeShapes.y = 1.;\n            rEyeShapes.x = 0.;\n            rEyeShapes.y = 1.;\n        }\n        else if (fract(iTime) &lt;= 2. * blinkStageDuration)\n        {\n            lEyeShapes.x = 1.;\n            lEyeShapes.y = 2.;\n            rEyeShapes.x = 1.;\n            rEyeShapes.y = 2.;\n        }\n        ...\n    }\n }<\/code><\/pre>\n<p>You can find the final source code find <a href=\"https:\/\/www.shadertoy.com\/view\/4dKcRG\">here<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Throughout my previous work in graphics, I haven&#8217;t really done too much related to animation: sliding spheres\/models around sure, but not animating a single rig. To amend that I decided I wanted to implement some kind of key frame-based tech<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-76","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/posts\/76","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/comments?post=76"}],"version-history":[{"count":13,"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/posts\/76\/revisions"}],"predecessor-version":[{"id":232,"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/posts\/76\/revisions\/232"}],"wp:attachment":[{"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/media?parent=76"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/categories?post=76"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mattrobertson.ca\/blog\/wp-json\/wp\/v2\/tags?post=76"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}