Dora

nullcon HackIM 2020

Written August 3rd, 2020

Dora

Unfortunately, I deleted my solution to this challenge (and a bunch of other challenges 😢) while trying to make room for new challenges. I need to be more proactive about storing these on writing solutions AND storing them on Github. However, this Dora challenge was a very interesting one, something I had never done before, and made me appreciate how important thinking outside of the box is, so I've written this solution.

The source code for the challenge is available on Github, so I could recreate my code, but I spent about 16 hours on this challenge and never want to spend that much that staring at Dora again.

The Premise

For this challenge, you connected to an open port and received the following message: "Where’s Dora? 1 for upper right, 2 for upper left, 3 for lower left, 4 for lower right". Then, a long base64 string was sent. When decoded, this string turned out to be a randomly generated image with a single-color background and characters from the Dora kids show.

To get the flag, you need to tell the server which corner Dora was. While I didn't know it at the time, to earn the flag you needed to this 800 times! Obviously, doing this by hand would be insane. So, I set out to automate it using Python and it's image recognition libraries.

Stumped

Boy, was I in over my head. The first thing to pop in my head was to use machine learning and image recognition (of which, I had no prior knowledge besides a graduate data mining course) to recognize dora. I used template matching at first using a simple cv2 guide. This was working out well at first, as the first five images I received all contained the exact same image of Dora. I tested out my script on these five images, and set it running. Poof! First Dora to come from that run was completely different: the first Dora was her and Boots dancing, and the new one was Dora as a mermaid! Template matching completely ruined.

Back to the drawing board. I wrote a quick script to just pull Dora down over and over, and found there were at least 6 different Doras (looking at the source, there were 10). Ok, Dora's face is pretty much the same in all of the images, so we will template match on just her face. I began doing so, taking excerpts of just Dora's face to use, and then noticed that the images of Dora were actually different sizes. Drat. Stumped again.

Starting from scratch, I started writing a template matching program that checked using various sizes, then taking the most confident solution (taking heavy inspiration from this tutorial). This worked fairly well, but would occasionally make mistakes. In addition, I realized that not only was Dora in different poses and sizes, but also different quality! Parts of her would be washed over by the background, making her look completely different. The images on the right from a write-up by Bechma shows this changing Dora off (their solution is very different from my final solution, and is definitely much more efficient).

I spent most of my time trying to get this to work by introducing more and more templates in gray scale. Obviously, I didn't get very far. With 800 images, it was almost certain my classifier would fail on at least one of them. At this point, I had been working on this challenge for about 12 hours, and gave up. I went to bed disappointed in myself.

Simple Dora image

A washed out Dora in a new pose

Finally Cracking the Code

Once I woke up, I was feeling rejuvenated but had no interest in seeing Dora again. While eating lunch, I pulled up a Dora picture I had sent to another club member, and the solution hit me. I raced home, pulled out my laptop, and confirmed my idea: Dora changes dramatically, but her friends are exactly the same! Using my template matching script, instead of matching Dora, I matched her friends, one after the other, and replaced them with the same color as the background. This would leave Dora all by herself on a single background.

With Dora singled out, the script simply then iterated through each pixel in the image from top to bottom until it found a new color. This almost worked; the template matching sometimes didn't match the full friend, leaving behind a pixel or two when the friends were deleted. Thus, the script was modified to find a 3x3 square of pixels that were different from the background. Once this square was found, the middle of Dora was roughly estimated and used to determine the corner Dora was in.

Letting the script run took quite a while, as my horrible code chugged along, but eventually it popped out the flag!

Page updated

Google Sites

Report abuse