Playing PokemonGo with folks wearing a Levis t-shirt

12 Aug 18 @18:00
Photo by Bruno Vaz for the amusing NIT article on people wearing Levis t-shirts.

All aboard the Levis shirt train! πŸ‘

If you are in downtown Lisbon, Porto or Algarve and happen to stroll around some crowded areas like shopping malls, there are moments where, at any given moment, you’ll literally have a Levis shirt in any of your eyesight's angle. There’s no escaping it!

Never mind the why (social identity, group belonging) or how (Instagram influencers?). The question isβ€Š—β€Šhow can we have some fun with this?

What if we had an app that recognises a Levis t-shirt and, each time you see one, you’d take a photo like PokemonGo and earn some points?

We could call it, like, Levimon (get it? πŸ™Š)

Defining the features

  1. Should be able to take photo with a phone
  2. Should be able to identify whether or not the person is wearing a Levis shirt
  3. Reward points to the player

Initially, my idea was to use the tracking.js library to track the red from the t-shirt, then isolate it in a square with a standard size using image manipulation library fabric.js.

We’d then use synaptic.js to build a Neural Network that could identify the brand.

In other words, a ton of work!First, you’d need to handle standardizing the photo to extract the brand area. Moreover and as importantly, you’d need to create a good training data set for the NN to identify the t-shirt.

That’s too f*ing long for a quick afternoon hackathon! Plus, there are some variations of it with blue shirts instead of white; some others look like a Pepsi logo. So, interesting challenge, but probably s*itty solution.

Hold on, how ‘bout using Clarifai? Those guys have some solid computer vision solutions, including identifying brands. On top of that, they have a free tier providing 5k transactions/month, ideal for our test case. Bingo! πŸ’°

*Actually* building the app

The app itself is a standard VueJS app created using vue-cli. The whole code can be found on Github.

Here are the interesting code nuggets:

Accessing the device’s camera

We do that by using the accept and capture parameters on a regular input element, such as:

// Within Shutter.vue component
<input id="cameraInput" type="file" name="camera" accept="image/*" class="image-input" capture @change="processImage($event)">

Reading the image & passing it to Clarifai

The Shutter component simply passes the whole event on input change. App.vue handles the rest. We call the identifyBrand method when imageData event is triggered.

//App.vue
<template>
<Shutter
  @imageData="identifyBrand"
/>
</template>

To read the content of the image we employ the FileReader API and use the onload callback to then feed the data back to Clarifai, such as:

<script>
const Clarifai = require('clarifai')
// Initialize Clarifai instance using your API key
const clarifai = new Clarifai.App({ apiKey: process.env.VUE_APP_CLARIFAI_KEY })
// Initialize our file reader
let reader = new FileReader()
...
methods:{
  identifyBrand(imageData){
    this.hasMatch = null
    let foundBrand = false
    reader.onload = (e) => {
      //  We only want to pass the Base64 encoded string to Clatifai
      let img = e.target.result.replace(/^data:image\/[a-z]+;base64,/, '')
      clarifai.models
      //  Use the right model to identify brands and pass the image as base64
      .predict(process.env.VUE_APP_PREDICT_MODEL, { base64: img })
      .then((r) => {
        if (r.status.code === 10000) {
          if (r.outputs.length > 0) {
            //  Check if Clarifai has returned any matches
            if (Object.keys(r.outputs[0].data).length > 0) {
              //  Clarifai returns the multiple places with matches in the image
              r.outputs[0].data.regions.map( el => {
                  
                //  Bulletproof. Just looking for 'levi' match in the name value
                if (el.data.concepts[0].name.toLowerCase().match('levi') !== null) {
                  //  Only count the match if the confidence score is above .9
                  if (el.data.concepts[0].value >= 0.9) {
                    foundBrand = true
                  }
                }
              })
              //  If there's a match, change this value to then display a success/error dialog
              if (foundBrand) {
                this.hasMatch = true
              } else {
                this.hasMatch = false
              }
            } else {
              this.hasMatch = false
            }
          }
        }
      },
      (err) => {
        alert(err)
      })
      .catch(err => {
        console.log(err)
      })
    }
    // Load imageData  
    reader.readAsDataURL(imageData)
  },
  
  ...
</script>

Testing it πŸ„

And there you go! A very rudimentary way of having some fun-with-friends: “Hey, who can spot the most Levimons out there?β€Š—β€ŠAh there’s a ton of ’em here!”.

Here’s the live test of the app:

After walking for 1h, I spotted at least 20 people wearing it! πŸ™ˆ

Most of them were too far or it was just to awkward to snap a picture hence a lot of missed opportunities.

Regarding the accuracy of Clarifai, unfortunately most of the times it wouldn’t get a match (#sad), even on very up-front & close pictures (lightning condition? Angle? Ripples in the shirt’s fabric?) so never mind my score..

Keep in mind though that this demo only keeps track of the score using Vuex, so no persistent storage. If time permits, I may have some fun in the future plugin in Firebase, just for the kicks of it.

Besides storing to a DB, some other nice-to-haves would be:

  • Streaming the camera image/video directly to Clarifai and have it more real-time;
  • Blur people’s faces automatically using tracking.js;
  • Adding some geolocation data to it;

Links