Building a Voice Controlled shopping assistant bot 🛒

14 Oct 16 @18:00

Jarbas is a voice powered (Portuguese only) groceries shopping assistant built for the Pixels Camp's 2016 Hackathon. 

Ever been in a situation where you notice that you're missing something in your house, you need to buy it but then forget to add that product to your shopping list?

Well now with Jarbas, you don't need to open your note app to write down your shopping list.

Simply say "Jarbas" to your phone (Android) or using the Jarbas Webapp (Chrome only) and let it help you note your shopping list. Jarbas will be listening to what you want to find and save the list for you. Once you're done, you can ask him to email you the list or request for the groceries to be delivered at your place (using Continente's API). Never forget that toilet paper anymore! Jarbas is here for you!

How it works

  • The bot runs on Chrome only (because of Googles Speech to Text API);
  • Constantly listening to the user's voice with microphone;
  • After the final transcript is returned from Google's API, it attempts to match commands using a bag of words approach;
  • Product matching is done using simple regex;
  • If products are found and returned, the bot waits for the user to state which product he wishes to add to his shopping list.
  • Product similarity is calculated using Sørensen–Dice's Coefficient;
  • Once the user is done with his purchase, he can ask for the list to be finalized; The list can be then sent to the user's email if he asks for it (ToDo);
  • Previously bought products can be found on the user's main page, including some stats like the most bought product and purchase mean cost;
  • Products were provided by Continente through their API. To speed up matching, products were stored on a Mongo database so the bot wouldn't constantly request the API to find/retrieve products;
  • An Android app was built by Miguel Pescadinha that loads the webapp into a webframe in order to use Android's native Speech to Text API; The native app and webapp talked to each other using Android Javascript Interface;


How it went

After 2 days of hacking away and testing, Jarbas was live (on now defunct Xervo (formerly known as Modulus).  You can check the presentation below.

The project landed 3rd place amongst over 60 participating teams 🏆.

Awesome trophy

You can check the code here.

Kudos for the organizing team of Pixels Camp and hopefully there'll be more next year!