This Snap! game uses Nvidia Jetson capability to classify images.

Please download and open Offline version of Snap! for our project. Go to https://snap.berkeley.edu/offline and follow the steps.

Please open the link Classification Game to download our project on your computer. Probably you would see the xml in raw format. Click the right button of your mouse and save it on the disk.

You can get picture from your web camera in Snap!.

• video capture block to enable video capturing.
• Change value of set video transparency block to 0 for clear image.
• video snap on stage block reports picture from stage.

If you have not imported it yet, please download jetson blocks and import it to your Snap! project.

• Use connect to Jetson url block to connect Jetson.

• Store the value of jetson_name in a variable.
• Store the value of connect to Jetson url block in a variable for later use.

Here we will send video snap on stage to Jetson for processing. Jetson will respond back class name, confidence value and class ID.

• Use get response from Jetson block to send image , and get class name and confidence value.
  • First input slot is for jetson variable that stores websocket data.
  • Second input slot is for costume you want to be classified by Nvidia Jetson.

This section will demonstrate how to handle response variable to access class name and confidence value.

• class name is the 2nd item of 1st item of response block.
• confidence value is 2nd item of 2nd item of response block.

  
  Multiply confidence value by 100 to get percentage of confidence .

Speech functionality is available as a library in Snap!. Select export libraries from settings then choose speech module .

• Use join block to create text like I am confidence percent sure, it is class name .
• Use speak and wait block to read text a loud.

Last step is adding loop for the game.

• Use repeat block and put script inside of it.

Please open the link Classification Game: Extended to download the extended version of our project on your computer.

Drag and drop the start video custom block to start using web camera.

Repeat the steps from Part 1 to connect to the Jetson Computer.

Use When flag clicked , variable setting and our custom blocks for initializing the game

In the game process we use when block to catch the click event from the stage and change the current page and broadcasting to tell the other blocks that game is starting.

Also we added additional blocks to resume the game after it was stopped.

We use familiar when block to listen to the start event. Then the block for receiving classification data is used in connection with analyze block Inside analyze block we compare the class name with preset class names of food, that our pets consume, and broadcast to them. In case if the detected object is not suitable for any of them, we run speak block to pronounce the name of the object.

In the script part of each sprite there are already blocks, responsible for handling their choice events. We use speak , play sound , animate blocks to animate the sprites.