HSRW EOLab Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
~~NOTOC~~

<WRAP center round todo 70%>
\\
**UNDER CONTINUOUS DEVELOPMENT**
</WRAP>

====== Let's plaiy! - AI at Schools (IP 17, WS 2021/22) ======

==== Design and create teaching material (brainware, software and hardware) for AI / ML / DL / CV in schools. ====

===== Introduction =====


Artificial intelligence is a prevailing technology entails both opportunities and risks to society and the environment. It is crucial for students to acquire these competences in school, enabling them to master the technology at an early age. This allows them to avoid being dominated by AI, to de-mystify AI and to develop the ability for critical reflection in assessing opportunities, applicability and limits of AI for problem solving.


The usability of many AI toolkits is continuously improving. They require less and less expert knowledge to utilise them effectively. This simplification can be taken further. The material we develop should be suited for children from 9 - 11 years. We have to hide the complexity of AI in the background and provide easy to use programming and user interfaces as frontends. An option we will investigate, is to use the open source graphical programming languages SNAP!, similar to Scratch, and MicroBlocks to control AI enabled embedded computers.


The training material to be created focuses on education for sustainable development and other applications oriented towards the common good. Concrete examples are biodiversity monitoring by detecting plant and animal species with robot-borne computer vision and deep learning.

We will develop a course curriculum and tutorial material (brainware), as well as hardware and software for AI-enabled small wheeled robots and tiny DIY drones to be used in education. The open course-ware will be provided and promoted appropriately. The ultimate goal is to design and implement course materials (brainware, hardware, software) for teaching AI in schools.

==== Initial Course Announcement ====

**{{ :ip:ws2021:lets_plaiy:ip2021-22_lets_plaiy_ai_for_schools_v003_rb.pdf |COURSE DESCRIPTION}}**

==== Short Description, German ====

[[.:Kurzbeschreibung:start]]

===== Final Project Poster =====

/*
https://wiki.eolab.de/lib/exe/fetch.php?media=ip:ws2021:lets_plaiy:student-documentation:lets_plaiy_ip17_ws2021_poster.pdf
*/

{{.:student-documentation:lets_plaiy_ip17_ws2021_poster.pdf}}



===== Inspiration  =====


|  {{ :ip:ws2021:lets_plaiy:cover.jpg?800 |}}  ||
|  Powered by NVIDIA Jetson Nano  ||
| {{youtube>RQWkkywzQF0?medium}}                                                                                                                                 | {{youtube>xk4wo7Yql-U?medium}}                                                                                  |
| **:!: 1. Introduction** Video by Harley Lara, \\  based on the {{ :ip:ws2021:lets_plaiy:ip2021-22_lets_plaiy_ai_for_schools_v003_rb.pdf |Course Description}}  | **2. SNAP** Jens Mönig demonstrates a simple object classification algorithm \\ for single stroke drawings.     |
| {{youtube>obt60r8ZeB0?medium}}                                                                                                                                 | {{https://teawiki.net/lib/exe/fetch.php?media=ip:ws2021:lets_plaiy:hsrw_jetbot_model_view_03.jpeg?400&direct}}  |
| **3. NVIDIA Jetson Nano Object Detection Inference**, \\ Dustin Franklin demonstrates detectnet on Jetson.                                                     | **4. Jetbot, HSRW Design**, \\ designed by Harley Lara                                                          |
| {{https://wiki.keyestudio.com/images/thumb/7/7d/Ks0356.png/320px-Ks0356.png}}                                                                                  | {{youtube>cf2xsYSTqgY?medium}}                                                                                  |
| ** 5. Micro:bit Robot Car ** \\ [[https://wiki.keyestudio.com/Ks0356_keyestudio_Micro:bit_Mini_Smart_Robot_Car|Keyestudio Micro:bit Mini Smart Robot Car]]     | **6. MicroBlocks** for Micro:bit                                                                                |

===== Distributed Architecture =====



The system is made of several interacting components which can be distributed across the hardware in different manners.


{{ :ip:ws2021:lets_plaiy:usage_scenarios.png?600 |}}




===== Schedule =====

----

^ Session  ^ Date                   ^ Location                ^ Topics                            ^ Intended Session Outcome                                                                                                                                                                                 ^ Lab Exercise                                                                                                                 ^ Session Log                                          ^ Assignments                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ^
|        1 | 30.09.2021             | IoT Lab \\ (02 02 510)  | Introduction                      | Jetsons assembled                                                                                                                                                                                        | Jetson Nano Assembly                                                                                                         | [[.:session_logs:start#section20210930|2021-09-30]]  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|        2 | 07.10.2021             | IoT Lab \\ (02 02 510)  | Software Installation on Jetsons  | Systems ready to use                                                                                                                                                                                     | [[.:lab:Basic Setup:start]] of NVIDIA Jetson Nano                                                                            | [[.:session_logs:start#section20211007|2021-10-07]]  | Learn Linux                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
|        3 | 14.10.2021             | IoT Lab \\ (02 02 510)  | Playing with object detection     | Practical experience how to run the OD docker container, insight into strengths and limitations of OD                                                                                                    | Prepare the Jetsons to run object detection (detectnet) with one of the CSI cams: \\ [[.:lab:Hello Object Detection:start]]  | [[.:session_logs:start#section20211014|2021-10-14]]  | Play with SNAP!                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|        4 | 21.10.2021             | IoT Lab \\ (02 02 510)  | Hello API from Snap!              | Practical experience how to get data from the EOLab Wheather Station API and display information into Snap!                                                                                              |  {{ :ip:ws2021:lets_plaiy:hello_api.zip | Hello API with Snap!}}                                                             | [[.:session_logs:start#section20211021|2021-10-21]]  | Develop proposals on how to communicate Snap! with a backend for image processing                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|        5 | 28.10.2021             | IoT Lab \\ (02 02 510)  | Snap! and backend communication   | Understand the communication system between Snap! and the Jetson Nano and communicate with the test server                                                                                               | Send base64-encoded image from Snap! to the Flask server running on the Jetson Nano and get the server's reply               | [[.:session_logs:start#section20211028|2021-10-28]]  | - Setup the Flask server on your own Jetson Nano \\ - Communicating a Snap! project with the Flask server \\ - Create a Demo in Snap! that could be interesting to use as didactic material, each student must present his or her demo in the next session.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
|        6 | 04.11.2021             | IoT Lab \\ (02 02 510)  | Presentation of project ideas     | Identify attractive project proposals to be implemented as didactic demonstrators in the learning material, in addition to providing support and guidelines to particular problems faced by each group.  | Each group presents progress in software setup in the Backend and communication with Snap.                                   | [[.:session_logs:start#section20211104|2021-11-04]]  | 1) Install Ilgar's and Harley's scripts on frontend (SNAP!) and backend (flask, python). \\ Send images from SNAP! to  the image classifier and receive the results. \\ 2) Create a presentation on your ideas of how to realize the training. \\ Several software architectures are possible. \\ **[[https://github.com/harleylara/lets-plaiy|Let's plaiy! Git repository]]**                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|        7 | 11.11.2021             | IoT Lab \\ (02 02 510)  |                                   |                                                                                                                                                                                                          |                                                                                                                              |                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|        8 | 18.11.2021             | IoT Lab \\ (02 02 510)  |                                   |                                                                                                                                                                                                          |                                                                                                                              | [[.:session_logs:start#section20211118|2021-11-18]]  | 1) **In group**: create a didactic project using Snap! and Nvidia AI models and **document it** on the **[[https://wiki.eolab.de/doku.php?id=ip:ws2021:lets_plaiy:student-documentation:start|Wiki]] or [[https://moodle.hochschule-rhein-waal.de/course/view.php?id=14457| Moodle]]**. The project should be replicable by children, and the material should serve as guide for the project. In the next session they should **present the project and the documentation**.\\ 2) **In group**: Propose using a diagram an architecture/workflow for the **training task (as detailed as possible)**, it should contain information flow, communication methods between components, interfaces etc. How can we manage the training? How can children take own photos, label them and use them for transfer learning?  |
|        9 | 25.11.2021             | IoT Lab \\ (02 02 510)  |                                   |                                                                                                                                                                                                          |                                                                                                                              |                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|       10 | 02.12.2021             | IoT Lab \\ (02 02 510)  |                                   |                                                                                                                                                                                                          |                                                                                                                              | [[.:session_logs:start#section20211202|2021-12-02]]  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|       11 | 09.12.2021             |                         |                                   |                                                                                                                                                                                                          |                                                                                                                              |                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|       12 | 16.12.2021             |                         |                                   |                                                                                                                                                                                                          |                                                                                                                              |                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|      --- | <del>23.12.2021</del>  |                         |                                   |                                                                                                                                                                                                          |                                                                                                                              |                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|      --- | <del>30.12.2021</del>  |                         |                                   |                                                                                                                                                                                                          |                                                                                                                              |                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|       13 | 06.01.2022             |                         |                                   |                                                                                                                                                                                                          |                                                                                                                              |                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|       14 | 13.01.2022             |                         |                                   |                                                                                                                                                                                                          |                                                                                                                              |                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|       15 | 20.01.2022             | Foyer of Audimax        | Poster Session                    |                                                                                                                                                                                                          |                                                                                                                              |                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 16       | 10.02.2020             | IoT Lab (02 02 510)     | Material evaluation               |                                                                                                                                                                                                          |                                                                                                                              | [[.:session_logs:start#section20220210|2022-02-10]]  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |





====== Work Packages ======
----

===== Software AI =====

  * Implement Object detection with DetectNet
  * Create a wrapper to communicate the neural network with Snap!

**Requirements:**
  * Modular Design
  * Good coding practices
  * Use standard communication protocols

===== Software Snap =====

  * Interfacing Snap (Front-end) with Jetson Nano (back-end)
    * WebSocket API ?
    * Web Serial API ?
    * Web USB API ?
    * Custom module inside Snap! ?

**Requirements:**
  * Modular Design
  * Good coding practices
  * Use standard communication protocols

===== Hardware Jetbots =====

  * Assemble prototype
  * Create assembly instructions
  * Electronics wiring diagrams

**Requirements:**
  * Easy assembly
  * Assembly using common tools
  * No specialized machinery
  * Modular design with the possibility to remove/place the Jetson Nano easily



====== Deliverables ======
----

**Main goal**: Develop, design and implement teaching materials on __artificial intelligence__ (AI) for schools (primary schools, age 9-11 years).

  * SNAP! course for kids
  * Jetbot assembly tutorial
  * Replicable projects using SNAP! as frontend:
    * Image classification
    * Predefined object detection with camera: (or rtp, rtsp, rtmp) with ssd-mobilenet
    * Control the Jetbot robot
    * Training convolutional neural network (CNN) for detection of new objects

<WRAP center round info 60%>
The complexity of all projects should be hidden, so all projects should be easy to implement using SNAP! as the interface.
</WRAP>

====== Deliverables, requirements ======
----

  * **Learning path (easy to follow)**: The teaching material should be easy to follow, with a defined structure that connects each of the topics in a way that is easy to absorb. The material can even have a story telling that proposes solutions to current problems such as global warming, garbage in the oceans, deforestation and more.
  * **Content**: The content of the material may be supported by different multimedia means such as videos, images, audio, etc. The way of presenting the content must be precise and easy to understand.
  * **Interactive**: The content must be interactive, with gamification strategies that encourage curiosity to continue learning.
  * **Learning experience**: The learning experience must be supported by proposed mini-projects, challenges, examples and a final project that unifies all that has been learned.
  * **Appropriate use of licenses**: The teaching material must be open and accessible to everybody under the license [[https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en|CC Attribution-Noncommercial-Share Alike 4.0 International]] so it is important to make use of resources (videos, audios, images and others) that have a license that allows copying, redistribution, modification and transformation, as far as possible __//you should generate your own content under the same license of the course//__.



====== Documentation ======


  * :!: **[[ip:ws2021:lets_plaiy:student-documentation:start|Student Documentation Space]]**

====== Internal Links ======

  * **[[.:todo_t:start|TEACHERS' TO DO]]**

  * [[.:Session Logs:start]]

 * [[https://wiki.eolab.de/doku.php?id=ip:ws2021:lets_plaiy:linux_cheat-sheet | linux]]

====== External Resources ======

  - {{https://computingeducation.de/pub/2019_Jatzlau-Michaeli-Seegerer-Romeike_BLOCKSANDBEYOND19.pdf|It’s not Magic After All - Machine Learning in Snap! using Reinforcement Learning}}, Sven Jatzlau et al.   
  - {{http://ceur-ws.org/Vol-2193/paper1.pdf|I Programming by Children using Snap! Block Programming in a Developing Country}}, \\ by Ken Kahn, Rani Megasari, Erna Piantari, and Enjun Junaeti
  - {{http://www.constructionismconf.org/wp-content/uploads/2020/05/C2020-Proceedings.pdf|Proceedings of the 2020 Constructionism Conference}}
  - {{https://youtu.be/p4JavDEA-q4 | Hyper Blocks in Snap! v6 }}
  - [[https://codebeautify.org/base64-to-image-converter|External Link]]