HSRW EOLab Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Image Classification Game: Part 1 ======
This ** Snap! game **  uses ** Nvidia Jetson ** capability to classify images.

===== Offline Snap! downloading =====

Please download and open Offline version of Snap! for our project. Go to [[https://snap.berkeley.edu/offline|https://snap.berkeley.edu/offline]] and follow the steps.

===== Snap! files' downloading =====

Please open the link [[https://github.com/EOLab-HSRW/Snapcon-workshop/blob/main/snap/snapcon_mlz_final_pre_project_base.xml|Classification Game]] to download our project on your computer. Probably you would see the xml in raw format. Click the right button of your mouse and save it on the disk.

{{ :snapcon2022:pic_4.jpg?600 |}}

===== Web camera Image in Snap! =====
You can get picture from your web camera in Snap!. 

  * ** video capture ** block to enable video capturing. {{ :snapcon2022:snap-video-capture.png?direct&200 |}}
  * Change value of ** set video transparency ** block to // 0 // for clear image. {{ :snapcon2022:snap-video-transparency.png?direct&200 |}}
  * ** video snap on stage ** block reports picture from stage.{{ :snapcon2022:snap-video-snap-stage.png?direct&200 |}} 

===== Connection to Jetson from Snap! =====
If you have not imported it yet, please download [[https://github.com/EOLab-HSRW/Snapcon-workshop/blob/main/snap/jetson_block.xml|jetson blocks]] and import it to your Snap! project. <WRAP center round info 40%> \\ 
Enable **Javascript Extensions** for following blocks.
</WRAP>

  * Use ** connect to Jetson url ** block to connect Jetson. {{ :snapcon2022:pic_1.png?300 |}}
<WRAP center round info 60%> \\
All participants need ** IP address ** of Nvidia Jetson in order to connect.
</WRAP>
  * Store the value of ** jetson_name ** in a variable. {{ :snapcon2022:pic_13.png?300 |}}
  * Store the value of ** connect to Jetson url ** block in a variable for later use.  {{ :snapcon2022:pic_3.png?400 |}}
===== Response from classification  =====
Here we will send **video snap on stage** to Jetson for processing. Jetson will respond back class name, confidence value and class ID. <WRAP center round info 60%> \\ 
Only **class name** and **confidence value** will be used in this example. This project does not use <del> **class ID** </del>.
</WRAP>
  * Use ** get response from Jetson ** block to send ** image **, and get **class name** and **confidence value**.
    * First input slot is for **jetson** variable that stores websocket data.
    * Second input slot is for **costume** you want to be classified by Nvidia Jetson.
{{ :snapcon2022:pic_4.png?400 |}}
===== Class name and confidence value  =====
This section will demonstrate how to handle ** response ** variable to access ** class name ** and ** confidence value. **
  * ** class name ** is the 2nd item of 1st item of ** response ** block. {{ :snapcon2022:snap-item-of-response.png?direct&400 |}} 
  * ** confidence value ** is 2nd item of 2nd item of ** response ** block.{{ :snapcon2022:snap-2item-response.png?direct&400 |}}<WRAP center round info 60%> \\
Multiply ** confidence ** value by 100 to get percentage of ** confidence **.</WRAP>

<WRAP center round info 60% > \\
You can create custom blocks, to get ** class name ** // get class name from response // and to get ** confidence ** // get confidence from response //. </WRAP> 

===== Speech functionality =====
** Speech functionality ** is available as a library in Snap!. Select // export libraries // from settings then choose // speech module //.
  * Use  ** join ** block to create text like // I am ** confidence ** percent sure, it is ** class name ** //.{{ :snapcon2022:snap-join-classname.png?direct&400 |}}
  * Use ** speak and wait ** block to read text a loud. {{ :snapcon2022:snap-speak-classname.png?direct&400 |}}

===== Repeat block for game =====
Last step is adding loop for the game.
  * Use repeat block and put script inside of it.
{{ :snapcon2022:pic_5.png?400 |}}
<WRAP center round info 60% > \\ 
This example used ** repeat until ** block to break loop when // space // key pressed. \\ 
You can download full game from [[https://github.com/EOLab-HSRW/Snapcon-workshop/blob/main/snap/snapcon_mlz_final_pre_project.xml | Github page of EOLab-HSRW.]] </WRAP> 

====== Image Classification Game: Part 2======

Please open the link [[https://github.com/EOLab-HSRW/Snapcon-workshop/blob/main/snap/snapcon_mlz_final_base.xml|Classification Game: Extended]] to download the extended version of our project on your computer.

===== Start Camera =====

Drag and drop the ** start video ** custom block to start using web camera.

{{ :snapcon2022:pic_6.png?100 |}}

===== Connection to Jetson =====

Repeat the steps from Part 1 to connect to the Jetson Computer.

{{ :snapcon2022:pic_13.png?300 |}}

{{ :snapcon2022:pic_3.png?400 |}}


===== Game Initialization =====
Use ** When flag clicked ** , variable setting and our custom blocks for initializing the game

{{ :snapcon2022:pic_7.png?300 |}}

===== Game process control blocks =====
In the game process we use ** when ** block to catch the click event from the stage and change the current page and broadcasting to tell the other blocks that game is starting.
{{ :snapcon2022:pic_8.png?300 |}}

Also we added additional blocks to resume the game after it was stopped.
{{ :snapcon2022:pic_9.png?300 |}}

===== Main part: sending images to Jetson and having fun with our pets! :-) =====

We use familiar ** when ** block to listen to the start event. Then the block for receiving classification data is used in connection with ** analyze ** block 
{{ :snapcon2022:pic10.png?400 |}}
Inside ** analyze ** block we compare the class name with preset class names of food, that our pets consume, and broadcast to them. In case if the detected object is not suitable for any of them, we run ** speak ** block to pronounce the name of the object.
{{ :snapcon2022:pic_11.png?400 |}}

In the script part of each sprite there are already blocks, responsible for handling their choice events. We use ** speak **, ** play sound **,** animate ** blocks to animate the sprites.
{{ :snapcon2022:pic_12.png?400 |}}

<WRAP center round info 60% > 
You can download full game from [[https://github.com/EOLab-HSRW/Snapcon-workshop/blob/main/snap/snapcon_mlz_final.xml | Github page of EOLab-HSRW.]] </WRAP>