Playing PokemonGo, only with folks wearing a Levis t-shirt

If you happen to stroll around Lisbon, Porto or Algarve around some crowded areas like shopping malls, there are moments where, at any given moment, you’ll literally have a Levis shirt in any of your eyesight's angle. There’s no escaping it!

All aboard the Levis shirt train!

So, what if we had an app that recognized a Levis t-shirt and, each time you saw one, you’d could take a photo like PokemonGo and earn some points?

We could call it Levimon (get it? 🙊)

Specs

1. Should be able to take photo with a phone
2. Should be able to identify whether or not the person is wearing a Levis shirt
3. Reward points to the player

Initially, my idea was to use the tracking.js library to track the red from the t-shirt, then isolate it in a square with a standard size using image manipulation library fabric.js.

We’d then use synaptic.js to build a Neural Network that could identify the brand.

In other words, a ton of work!

First, you’d need to handle standardizing the photo to extract the brand area. Moreover and as importantly, you’d need to create a good training data set for the NN to identify the t-shirt.

App prototype.

That’s too much for an afternoon project! Plus, there are some variations of it with blue shirts instead of white; some others look like a Pepsi logo. So, interesting challenge, but probably crappy solution.

Hold on, how ‘bout using Clarifai? Those guys have some solid computer vision solutions, including identifying brands. On top of that, they have a free tier providing 5k transactions/month, ideal for our test case. Bingo!

Actually building the app

The app itself is a standard VueJS app created using vue-cli. The whole code can be found on GitHub. Below are some of the interesting code nuggets.

Accessing the device’s camera

We do that by using the accept and capture parameters on a regular input element, such as:

Reading the image & passing it to Clarifai

The Shutter component simply passes the whole event on input change. App.vue handles the rest. We call the identifyBrand method when imageData event is triggered.

Then, to read the content of the image we employ the FileReader API and use the onload callback to then feed the data back to Clarifai, such as:

...
methods:{
  identifyBrand(imageData){
    this.hasMatch = null
    let foundBrand = false
    reader.onload = (e) => {
      //  We only want to pass the Base64 encoded string to Clatifai
      let img = e.target.result.replace(/^data:image\/\[a-z]+;base64,/, '')
      clarifai.models
      //  Use the right model to identify brands and pass the image as base64
      .predict(process.env.VUE_APP_PREDICT_MODEL, { base64: img })
      .then((r) => {
        if (r.status.code === 10000) {
          if (r.outputs.length > 0) {
            //  Check if Clarifai has returned any matches
            if (Object.keys(r.outputs\[0].data).length > 0) {
              //  Clarifai returns the multiple places with matches in the image
              r.outputs\[0].data.regions.map( el => {
                //  Bulletproof. Just looking for 'levi' match in the name value
                if (el.data.concepts\[0].name.toLowerCase().match('levi') !== null) {
                  //  Only count the match if the confidence score is above .9
                  if (el.data.concepts\[0].value >= 0.9) {
                    foundBrand = true
                  }
                }
              })
              //  If there's a match, change this value to then display a success/error dialog
              if (foundBrand) {
                this.hasMatch = true
              } else {
                this.hasMatch = false
              }
            } else {
              this.hasMatch = false
            }
          }
        }
      },
      (err) => {
        alert(err)
      })
      .catch(err => {
        console.log(err)
      })
    }
    // Load imageData
    reader.readAsDataURL(imageData)
  },
  ...

Testing In The Wild

Photo by Bruno Vaz for the amusing NIT article on people wearing Levis t-shirts.

After walking for 1h, I spotted at least 20 people wearing it! 🙈

Most of them were too far or it was just to awkward to snap a picture hence a lot of missed opportunities.

Regarding the accuracy of Clarifai, unfortunately most of the times it wouldn’t get a match, even on very up-front & close pictures (Lightning conditions? Angle? Ripples in the shirt’s fabric?) so never mind my score..

Keep in mind though that this demo only keeps track of the score using Vuex, so no persistent storage. If time permits, I may have some fun in the future plugin in Firebase, just for the kicks of it.

Besides storing to a DB, some other nice-to-haves would be:

• Streaming the camera image/video directly to Clarifai and have it more real-time;
• Blur people’s faces automatically using tracking.js;
• Adding some geolocation data to it?;

And there you go! A very rudimentary way of having some fun-with-friends: "Hey, who can spot the most Levimons out there? — Ah there’s a ton of ’em here!"

Try it out here. Get the code here.