Objective

This tutorial will guide you through the process of setting up and running a pre-built keyword spotter on a SAME54 using a model developed with Edge Impulse. This tutorial includes the training data, trained model, and deployment code so you can get up and running quickly, but I’ll also be explaining how you can modify the project to develop your own keyword spotter.

Back to top.

Materials

Hardware Tools

• Microchip Technology SAM E54 Curiosity Ultra Development Board

• WM8904 Audio Codec Module

• Analog microphone with 3.5mm connector

Back to top.

Software Tools

• MPLAB® X IDE
• MPLAB IPE
• Edge Impulse Studio (Free account; No download required)

Back to top.

Exercise Files

• The firmware, MPLAB X IDE project, and instructions for generating your own keyword dataset can be found in the GitHub repository.
• Pre-built "up" and "down" datasets can be downloaded from the latest GitHub release.
• Pre-built firmware for "yes" and "no" Keyword spotters can be downloaded from the latest GitHub release.

Back to top.

Procedure

Before we get started, you'll need to install and set up the required software as detailed in the steps below.

Back to Top

Configuring the Hardware

To enable audio collection for the SAME54, we first need to install the WM8904 daughterboard and configure the board’s jumpers appropriately. We’ll use Figure 2 (taken from the Users Guide) as a map for the different components on the board.

1. Connect the WM8904 daughterboard to the X32 audio interface (labeled 5 above) making sure to orient the board so that the audio connectors face the edge of the board.
2. Set the jumpers on the audio codec board to match Figure 3; each jumper should be connected to the two left-most pins on the board when viewed from the perspective of the audio jacks.

3. Set the CLK SELECT jumper (labeled 10 above) so that it connects the MCLK and PA17 pins on the board as shown in Figure 4. This pin configuration lets the WM8904 act as the audio clock master to the SAME54’s I2S peripheral.

4. Connect your microphone to the audio daughterboard’s MIC IN jack.

5. Connect your PC’s USB connection to the Curiosity board’s EDBG micro USB connector (labeled 2 above).

Great! The hardware is now configured for the keyword spot demo project. If you have MPLAB X IDE open, the Curiosity board should be automatically detected by MPLAB X IDE.

For more detailed information about the Curiosity Ultra board including schematics, consult the user guide.

Back to top.

Keyword Spotter Firmware Overview

Before we get started, you'll need to install and set up the required software as detailed in the steps below.

We’re now set up to run the pre-built firmware. Go ahead and program your device with the firmware HEX file from the latest GitHub release using the MPLAB IPE tool.

To present the keyword detection status, the firmware flashes the onboard LED1 and LED2 according to Table 1. The LEDs can be located at the top left of the development board near the barrel connector; see Figure 5 for reference.

The LEDs will light up selectively based on which keyword is detected. The Program status LED states table summarizes the behavior.

Table 1: Program status LED states

In addition, the firmware also prints the confidence numbers for each class over the UART port. To read the UART port use a terminal emulator of your choice (e.g., PuTTY for Windows) with the following settings:

• Baudrate 115200
• Data bits 8
• Stop bits 1
• Parity None

For reference, an example of the output is shown in Figure 6. Notice that the confidence numbers are between 0 (no confidence) and 1 (certainty) and the class confidences roughly sum up to 1 (there is some arithmetic error that makes the sum slightly less than 1).

Back to Top

Going Further: Changing Keywords

Okay, now you have a complete working example, but you might be wondering how you can modify the keyword spotter for your own needs. This section will cover the steps required to modify, re-train, and deploy the Edge Impulse project for a new task: recognizing the commands "up" and "down".

Back to top.

Cloning the Keyword Spotter Impulse

I’ve made my Edge Impulse project available publicly so that you can clone it and make any modifications you like. Public Edge Impulse projects are cloneable similar to how you might clone a git repository but instead of code, the cloned project will include all the labeled audio data, the neural network model, and all of the project’s configuration parameters. Follow the steps below to clone the project.

1. Open your browser and log in to your Edge Impulse account.

2. Browse to the keywordspotter_demo project.

3. Click the Clone this project box at the top right of the screen.

4. Wait for cloning to complete (may take several minutes).

You now have the project cloned to your account enabling you to make any changes you like.

Back to Top

Creating Your Dataset

Rather than collecting our own dataset, we'll be using the Google Speech Commands dataset (v0.02), and some background noise clips that are already included with the Edge Impulse keyword spotting dataset. A pre-built dataset for "up" "down" commands is included in the latest GitHub release, but there are instructions in the GitHub repository if you want to create your own.

1. Download the "up" and "down" keyword spotting dataset from the latest Github release page and extract it into a temporary folder.

2. Upload the dataset to your Edge Impulse keyword spotter project with the Edge Impulse Uploader tool. Note that we're letting Edge Impulse automatically select the training and test split of the data (this corresponds to the —category split option).

$ edge-impulse-uploader --clean
$ edge-impulse-uploader --label up --category split up/*.wav
$ edge-impulse-uploader --label down --category split down/*.wav
$ edge-impulse-uploader --label noise --category split noise/*.wav
$ edge-impulse-uploader --label unknown --category split unknown/*.wav

Back to Top

Feature Generation

With the data uploaded, we now need to perform the MFCC feature extraction step; this will generate the final form of the data to be used for training the model.

1. Click the MFCC item in the Edge Impulse Studio side menu.
2.  On the MFCC page, click the Generate features tab, and find and click the Generate features button.

Back to Top

Training Your Model

Now that the MFCC features for the input data have been generated, we can train the neural network.

Note that we are training "from scratch" here rather than using the Retrain model functionality; this is primarily done because we've changed the train and test data split (we don't want to evaluate performance with data we've already trained on), and secondarily because we are changing the class assignment of some of the data ("yes" "no" "unknown") so that transfer learning is unlikely to help us in this context.

Follow the steps below to train the model.

1. Click the NN classifier item in the Edge Impulse Studio side menu.
2. On the NN classifier page, click the Start training button to train the network.

The training will take several minutes. Once completed, Edge Impulse Studio will display a summary of the model's performance and size.

Back to Top

Deploying Your Impulse

When you’re satisfied with your trained model, follow the steps below to deploy the model and integrate it into your existing MPLAB X IDE project:

1. Switch to the Deployment tab in Edge Impulse Studio and select the C/C++ deployment option.

2. Click the Build button that appears at the bottom of the page, leaving the default options selected.

3. Unzip the contents of the edge impulse ZIP file into your MPLAB X project’s src/ folder so that they overwrite the original Edge Impulse SDK files.

4. Rename all the .cc files from the Edge Impulse library to have a CPP suffix. You can do this in one shot with the following commands:

a. On Windows^®: ren *.cc *.cpp

b. On Mac^®/Linux^®: find . -name "*.cc" -exec sh -c 'mv "$1" "${1%.cc}.cpp"' _ {} \;

If you're using the MPLAB X IDE keyword spotter project from this post as a base you shouldn't have to do any modification of the project configuration to use the new code. However, there is a little code in main.cpp that is specific to using the "yes" and "no" keywords which we modify in the next section.

Back to Top

Modify main.cpp

Last step; we just need to make a minor modification to the main program file to light the board LEDs when "up" and "down" are detected instead of "yes" and "no" (which are no longer defined as a class in the Edge Impulse code).

1. Open the keyword spotter MPLAB X IDE project, and bring up the main.cpp source file.
2. Inside the main while loop look for the literal strings "yes" and "no" and replace them with "up" and "down" respectively. See Figure 11 for reference.

Back to Top

Flash the Firmware

Okay, modifications to the project are complete, so click Make and Program Device in the toolbar ( icon) to program the device with the modified keyword spotter firmware. Once programming finishes, your board should respond to the keywords "up" and "down" (and reject the keywords "yes" and "no"). Copy and paste this mark up onto your page in source mode once for each side-by-side step you have. Once pasted in you can add content to the columns in WYSIWYG mode.

Back to Top

Final Remarks

That's it! You're now ready to start building your own keyword-spotting application.

For more details about integrating your Impulse with an existing MPLAB X IDE project, check out our "Integrating the Edge Impulse Inferencing SDK" article.

To learn more about Edge Impulse Studio, including tutorials for other machine learning applications, go to the Edge Impulse Docs.

Back to Top