Create a Smartbell with Edge Impulse
Objective
Turn your dumbbell into a smartbell with the SAMD21 Machine Learning Evaluation Kit and an embedded ML classifier built in the Edge Impulse Studio.
With the Edge Impulse Studio, it is now easier than ever to develop machine learning solutions that can be deployed on Microchip Arm® Cortex®-based 32-bit microcontrollers and microprocessors. This example will guide you through the process of collecting 6-axis IMU data from the SAMD21 Machine Learning Evaluation Kit, uploading the collected data to the Edge Impulse Studio, creating a custom impulse that can classify the input data, and finally deploying the impulse back to the Machine Learning Evaluation Kit.
See the table below to get an idea of the size and performance of the Smartbell application.
| Build Parameters | ROM Size | RAM Size | Inference Time |
|---|---|---|---|
| -O2 optimizations, 48MHz Clock | ~104kB | ~9kB | ~379ms (DSP: 369ms, NN: 1ms, Anomaly: 9ms) |
Materials
Hardware Tools
- SAMD21 Machine Learning Evaluation Kit with BOSCH IMU or with TDK IMU
- Dumbbell
- Choose a weight that you are comfortable with
- Micro USB cable
- At least 4 ft long to allow for performing exercises while connected
- Double-sided sticky tape
- Rubber bands
Software Tools
Exercise Files
- Data Collection Firmware:
- Bosch or TDK sensors are supported
- Impulse Deployment Firmware:
- Bosch sensor is supported
Connection Diagram
SAMD21 ML Evaluation Kit
The IMU click board is connected to the mikroBUS™ socket. Check the pin labels when connecting the boards to ensure that they are connected properly.
Dumbbell Mounting
The kit is mounted on one end of the dumbbell using double-sided sticky tape and rubber bands. The USB cable is tied around the handle to prevent the kit from being pulled off.
Procedure
Before getting started, be sure to install all the necessary software tools. We will work within MPLAB X IDE and build the project with the MPLAB XC32 compiler. Both MPLAB DV and the Machine Learning Plugin are available within the plugin update center in MPLAB X IDE. You will also need to sign up for a free account on the Edge Impulse Studio and then create a new project so that you can begin uploading training data.
Data Collection
The first step in building a machine-learning model is collecting data. This data will be used to train the ML model so that it can recognize different types of exercises. In this example, we will train the model to recognize three exercises.
To begin collecting data, the SAMD21 ML Kit must be programmed with the datastreamer firmware. Download the example project (Smartbell_Edge_Impulse.X) and open it in MPLAB X IDE. Check that the DATA_STREAMER_BUILD flag is set to 1, and then click Make and Program Device.
Next, open MPLAB Data Visualizer and load the workspace found within the downloaded example project (mplab-dv-workspace-6dof-imu-data.json). Be sure that the DGI connection for the SAMD21-IoT WG is disabled. Select the appropriate Serial/CDC Connection for the SAMD21, adjust the Baud Rate to 115200, and then click Apply. To start parsing the serial data, click the Play button.
Navigate to the Variable Streamers tab and select the Serial/CDC Connection as the input for the variable streamer.
You will now see the 6-axis IMU data plotted in the graph.
Now that the data is available in MPLAB Data Visualizer, you are ready to start collecting samples to train a machine learning model. It is best to have at least a few minutes of data for each type of exercise. Begin by performing a few repetitions of one exercise, and then double-click the graph in the Time Plot to stop it from scrolling. Fit the region of data corresponding to the exercise within the graph window and then click Mark. This will set the cursors at the boundaries of the window to mark that region for upload to the Edge Impulse Studio.
Next, log in with your Edge Impulse credentials within the ML Partners Plugin to configure the data upload. Select ax, ay, az, gx, gy, and gz as the Data Sources. In the Project field, select the Edge Impulse Studio project that the data will be uploaded to, and select Training as the Data Endpoint.
When configuration is complete, click Upload Data, and then open the Edge Impulse Studio within your browser. Go to the Data acquisition tab to see the uploaded data.
Follow this process for uploading data samples of each exercise until you have at least a few minutes of data for each. The samples can be of any length because the Edge Impulse Studio will handle slicing the data into windows of equal size. Collect data on each exercise performed by the right arm and the left arm. The model can only learn from what it will see in the training data, so it is important to build a dataset that is representative of the actual data that the model will see in deployment. If the dataset only contains data from one person, the model will learn the exact movement patterns of that person. If the dataset contains data from many people, the model will learn the general characteristics of each exercise that exist across different people.
Impulse Creation
Once you have a training dataset, navigate to the Impulse design tab where you can configure the window size of the data input into the impulse and select the processing blocks and learning blocks that the impulse will include. For this continuous motion recognition problem, we will use a Spectral Analysis block, a Neural Network block, and an Anomaly Detection block. Once the blocks have been added and the time series data has been configured, click the Save Impulse button. Start with a window size of 2000 ms and a window increase of 250 ms.
Time Series Data
The first block in the impulse is always the input data. The Edge Impulse Studio uses a sliding window to go over the raw data and slice it into samples of equal length. The Window increase is the offset used to shift the window between each snapshot. When the Window increase is less than the Window size, the sliding window will create a larger number of samples that overlap somewhat yet are still unique.
The Window size setting will depend on the length of the event that the model will learn to recognize. For periodic motion data like repetitions of a dumbbell exercise, the window size should include at least one complete performance of the exercise. It is a design trade-off between speed and accuracy. If the model is performing very well, then it may be possible to decrease the window size and thereby decrease the latency of the system while maintaining an acceptable level of accuracy.
Spectral Analysis
The data samples sliced by the sliding window will be passed into the Spectral Analysis block for pre-processing and feature extraction. The extracted features will be the actual data input into the learning blocks. Open the Spectral Features tab to configure the data pre-processing parameters.
The Spectral Analysis block allows for scaling, filtering, and spectral power analysis. The graph at the top shows the loaded data file. Drag the window to inspect different sample windows within the file. The graphs on the right show the output of the Spectral Analysis block to help with parameter tuning.
For the smartbell example project, the parameters are configured as shown in the above image. When you are ready to move on to the next step, click Save Parameters.
Next, click Generate Features and use the feature explorer to inspect the generated features. Each data sample will be colored in the graph according to its label. Use the drop-down box for each axis to plot the samples based on the different features. It is a good sign when the data of different labels are well separated in multiple dimensions.
Neural Network (Keras)
The neural network classifies each data sample by outputting a confidence score for each of the labels present in the training dataset. The input to the network is the feature array that is output from the Spectral Analysis block. The network type and structure can be configured in the NN Classifier menu.
For this example project, the Number of training cycles was set to 22 and the Learning rate was set to 0.001. The general rule in adjusting these two parameters is to find a sweet spot where the model achieves high accuracy by steadily improving with each training cycle. The Minimum confidence rating is simply a threshold for trusting the neural network result.
The hidden layers of the network can be edited in the Neural network architecture menu. There are many types of layers available, but for this example, all that is required are two fully connected dense layers. The latency and memory requirements can be minimized by selecting the simplest model that provides acceptable accuracy.
The Edge Impulse Studio enables you to experiment with different network architectures and quickly evaluate their performance to select the optimal network for any particular problem. When the network is performing well, you can move on to training the anomaly detection block.
K-Means Anomaly Detection
The K-Means Anomaly Detection layer is used in parallel with the neural network classifier to detect data that is significantly different from data found in the training dataset. This is generally useful as an indication of the reliability of the neural network. In this example, it will also serve as an indication of the correctness of the exercise being performed with the smartbell. For instance, if the exercise is performed too fast or with incorrect form, then the sample would be flagged as anomalous.
The k-means parameters can be adjusted in the Anomaly detection settings and the axes that will be used as input can be selected there as well. The RMS values for each axis are the recommended input when using the Spectral Analysis block. The cluster count parameter can be increased if the distribution of data across the generated features is more complex so more clusters can be used to approximate the distribution.
After the K-Means Anomaly Detection block is trained, you can move on to testing the impulse performance.
Impulse Testing
Once the neural network and anomaly detection layers have been trained, the performance can be tested by uploading new data to the testing endpoint. The same data upload process is used except that the Data Endpoint must be changed to Testing. Testing data is kept separate from training data so that the model performance can be verified with data that the model has not seen during training.
Live Classification
If you have the Live Classification tab open in the Edge Impulse Studio, it will automatically display the classification results for new data uploaded to the testing endpoint. Live Classification gives a detailed classification result for each sample and provides tools for inspecting each tested data sample within the dataset.
The Live Classification tool will slice uploaded data into sample windows using the window size and offset from the Impulse Design step. Then the sample windows are passed through the impulse for classification.
Model Testing
The Model Testing tool can be used to test model performance over the entire test dataset or batches of test data as needed. It gives an accuracy score based on the expected outcome for each test data file.
Model testing is useful for scoring the overall performance of the impulse and for identifying any samples that might require a closer look in the Live Classification tool.
Impulse Deployment
Once the impulse design and testing are complete, go to the Deployment tab to download the source code that can be added to your MPLAB X IDE project. Select the C++ Library option and then scroll down to select optimizations.
The Edge Impulse EON compiler greatly reduces the memory requirements and has no negative impacts on performance, so it is activated by default. There is also an option to quantize the NN to 8-bit values, which can save some ROM at the expense of RAM. For this example, 8-bit quantization is used because the difference in memory usage is trivial and the impulse actually has a slightly higher performance when quantized. When the desired optimizations are selected, click Build to download the source code for the impulse.
Firmware Integration
To quickly integrate a new impulse for the SAMD21 ML Kit, the example firmware can be used. Simply open the src folder within the example firmware, Smartbell-Edge-Impulse.X, and then replace the model-parameters and tflite-model folders with the folders of the same name from the newly downloaded source code. Set DATA_STREAMER_BUILD equal to 0 at the top of main.cpp and then program the SAMD21 ML Kit.
Viewing Embedded Classification Results
Once the device is programmed, leave it connected to the PC via USB and open the MPLAB Data Visualizer to display the classification results. Select the appropriate Serial/CDC Connection for the SAMD21, adjust the Baud rate to 115200, and then click Apply. Click play on the configured Serial/CDC Connection, and then open the Terminal and select the Serial/CDC Connection from the drop-down box.
The impulse deployment firmware collects IMU data in two-second windows and then passes it through the impulse for classification. The embedded classification results are then transmitted over UART for display in the terminal.
Conclusions
Now that you have seen how easy it is to build and deploy an embedded machine learning classifier with the SAMD21 ML Eval Kit and Edge Impulse, you are ready to add embedded classification and anomaly detection to your application. The SAMD21 ML Kit can support a variety of Sensor Click boards™ from MikroE, so get creative and explore the possibilities of embedded ML with Edge Impulse.