Note

Click here to download the full example code

DS-CNN/KWS inference

This tutorial illustrates how to build a basic speech recognition Akida network that recognizes thirty-two different words.

The model will be first defined as a CNN and trained in Keras, then converted using the CNN2SNN toolkit.

This example uses a Keyword Spotting Dataset prepared using TensorFlow audio recognition example utils.

The words to recognize are first converted to spectrogram images that allows us to use a model architecture that is typically used for image recognition tasks.

1. Load the preprocessed dataset

The TensorFlow speech_commands dataset is used for training and validation. All keywords except “backward”, “follow” and “forward”, are retrieved. These three words are kept to illustrate the edge learning in this edge example. The data are not directly used for training. They are preprocessed, transforming the audio files into MFCC features, well-suited for CNN networks. A pickle file containing the preprocessed data is available on our data server.

import pickle

from tensorflow.keras.utils import get_file

# Fetch pre-processed data for 32 keywords
fname = get_file(
    fname='kws_preprocessed_all_words_except_backward_follow_forward.pkl',
    origin="http://data.brainchip.com/dataset-mirror/kws/kws_preprocessed_all_words_except_backward_follow_forward.pkl",
    cache_subdir='datasets/kws')
with open(fname, 'rb') as f:
    [_, _, x_valid, y_valid, _, _, word_to_index, _] = pickle.load(f)

# Preprocessed dataset parameters
num_classes = len(word_to_index)

print("Wanted words and labels:\n", word_to_index)

Downloading data from http://data.brainchip.com/dataset-mirror/kws/kws_preprocessed_all_words_except_backward_follow_forward.pkl

    8192/62628765 [..............................] - ETA: 0s
  212992/62628765 [..............................] - ETA: 14s
  499712/62628765 [..............................] - ETA: 12s
  843776/62628765 [..............................] - ETA: 12s
 1196032/62628765 [..............................] - ETA: 11s
 1458176/62628765 [..............................] - ETA: 11s
 1720320/62628765 [..............................] - ETA: 11s
 1982464/62628765 [..............................] - ETA: 11s
 2260992/62628765 [>.............................] - ETA: 11s
 2588672/62628765 [>.............................] - ETA: 10s
 2932736/62628765 [>.............................] - ETA: 10s
 3219456/62628765 [>.............................] - ETA: 10s
 3457024/62628765 [>.............................] - ETA: 10s
 3612672/62628765 [>.............................] - ETA: 11s
 3776512/62628765 [>.............................] - ETA: 11s
 3940352/62628765 [>.............................] - ETA: 11s
 4136960/62628765 [>.............................] - ETA: 11s
 4349952/62628765 [=>............................] - ETA: 11s
 4579328/62628765 [=>............................] - ETA: 11s
 4808704/62628765 [=>............................] - ETA: 11s
 4915200/62628765 [=>............................] - ETA: 12s
 5062656/62628765 [=>............................] - ETA: 12s
 5185536/62628765 [=>............................] - ETA: 12s
 5308416/62628765 [=>............................] - ETA: 13s
 5447680/62628765 [=>............................] - ETA: 13s
 5603328/62628765 [=>............................] - ETA: 13s
 5742592/62628765 [=>............................] - ETA: 13s
 5914624/62628765 [=>............................] - ETA: 13s
 6086656/62628765 [=>............................] - ETA: 13s
 6258688/62628765 [=>............................] - ETA: 13s
 6463488/62628765 [==>...........................] - ETA: 13s
 6684672/62628765 [==>...........................] - ETA: 13s
 6922240/62628765 [==>...........................] - ETA: 13s
 7159808/62628765 [==>...........................] - ETA: 13s
 7413760/62628765 [==>...........................] - ETA: 13s
 7634944/62628765 [==>...........................] - ETA: 13s
 7905280/62628765 [==>...........................] - ETA: 13s
 8110080/62628765 [==>...........................] - ETA: 13s
 8404992/62628765 [===>..........................] - ETA: 12s
 8716288/62628765 [===>..........................] - ETA: 12s
 9003008/62628765 [===>..........................] - ETA: 12s
 9306112/62628765 [===>..........................] - ETA: 12s
 9576448/62628765 [===>..........................] - ETA: 12s
 9715712/62628765 [===>..........................] - ETA: 12s
 9912320/62628765 [===>..........................] - ETA: 12s
10067968/62628765 [===>..........................] - ETA: 12s
10231808/62628765 [===>..........................] - ETA: 12s
10412032/62628765 [===>..........................] - ETA: 12s
10625024/62628765 [====>.........................] - ETA: 12s
10821632/62628765 [====>.........................] - ETA: 12s
11010048/62628765 [====>.........................] - ETA: 12s
11173888/62628765 [====>.........................] - ETA: 12s
11337728/62628765 [====>.........................] - ETA: 12s
11509760/62628765 [====>.........................] - ETA: 12s
11698176/62628765 [====>.........................] - ETA: 12s
11837440/62628765 [====>.........................] - ETA: 12s
11976704/62628765 [====>.........................] - ETA: 12s
12132352/62628765 [====>.........................] - ETA: 12s
12304384/62628765 [====>.........................] - ETA: 12s
12402688/62628765 [====>.........................] - ETA: 12s
12517376/62628765 [====>.........................] - ETA: 12s
12615680/62628765 [=====>........................] - ETA: 12s
12722176/62628765 [=====>........................] - ETA: 12s
12836864/62628765 [=====>........................] - ETA: 13s
12959744/62628765 [=====>........................] - ETA: 13s
13099008/62628765 [=====>........................] - ETA: 13s
13230080/62628765 [=====>........................] - ETA: 13s
13385728/62628765 [=====>........................] - ETA: 13s
13541376/62628765 [=====>........................] - ETA: 13s
13713408/62628765 [=====>........................] - ETA: 13s
13910016/62628765 [=====>........................] - ETA: 13s
14123008/62628765 [=====>........................] - ETA: 12s
14336000/62628765 [=====>........................] - ETA: 12s
14565376/62628765 [=====>........................] - ETA: 12s
14778368/62628765 [======>.......................] - ETA: 12s
15032320/62628765 [======>.......................] - ETA: 12s
15253504/62628765 [======>.......................] - ETA: 12s
15515648/62628765 [======>.......................] - ETA: 12s
15761408/62628765 [======>.......................] - ETA: 12s
16023552/62628765 [======>.......................] - ETA: 12s
16318464/62628765 [======>.......................] - ETA: 12s
16588800/62628765 [======>.......................] - ETA: 11s
16883712/62628765 [=======>......................] - ETA: 11s
17178624/62628765 [=======>......................] - ETA: 11s
17301504/62628765 [=======>......................] - ETA: 11s
17473536/62628765 [=======>......................] - ETA: 11s
17629184/62628765 [=======>......................] - ETA: 11s
17801216/62628765 [=======>......................] - ETA: 11s
17981440/62628765 [=======>......................] - ETA: 11s
18161664/62628765 [=======>......................] - ETA: 11s
18358272/62628765 [=======>......................] - ETA: 11s
18563072/62628765 [=======>......................] - ETA: 11s
18776064/62628765 [=======>......................] - ETA: 11s
19005440/62628765 [========>.....................] - ETA: 11s
19234816/62628765 [========>.....................] - ETA: 11s
19472384/62628765 [========>.....................] - ETA: 11s
19693568/62628765 [========>.....................] - ETA: 11s
19816448/62628765 [========>.....................] - ETA: 11s
19972096/62628765 [========>.....................] - ETA: 11s
20111360/62628765 [========>.....................] - ETA: 11s
20267008/62628765 [========>.....................] - ETA: 11s
20414464/62628765 [========>.....................] - ETA: 11s
20578304/62628765 [========>.....................] - ETA: 11s
20758528/62628765 [========>.....................] - ETA: 10s
20938752/62628765 [=========>....................] - ETA: 10s
21143552/62628765 [=========>....................] - ETA: 10s
21348352/62628765 [=========>....................] - ETA: 10s
21536768/62628765 [=========>....................] - ETA: 10s
21766144/62628765 [=========>....................] - ETA: 10s
22020096/62628765 [=========>....................] - ETA: 10s
22282240/62628765 [=========>....................] - ETA: 10s
22503424/62628765 [=========>....................] - ETA: 10s
22773760/62628765 [=========>....................] - ETA: 10s
23011328/62628765 [==========>...................] - ETA: 10s
23265280/62628765 [==========>...................] - ETA: 10s
23543808/62628765 [==========>...................] - ETA: 10s
23863296/62628765 [==========>...................] - ETA: 9s 
24199168/62628765 [==========>...................] - ETA: 9s
24551424/62628765 [==========>...................] - ETA: 9s
24879104/62628765 [==========>...................] - ETA: 9s
25206784/62628765 [===========>..................] - ETA: 9s
25559040/62628765 [===========>..................] - ETA: 9s
25903104/62628765 [===========>..................] - ETA: 9s
26271744/62628765 [===========>..................] - ETA: 9s
26632192/62628765 [===========>..................] - ETA: 8s
26943488/62628765 [===========>..................] - ETA: 8s
27181056/62628765 [============>.................] - ETA: 8s
27320320/62628765 [============>.................] - ETA: 8s
27459584/62628765 [============>.................] - ETA: 8s
27615232/62628765 [============>.................] - ETA: 8s
27770880/62628765 [============>.................] - ETA: 8s
27942912/62628765 [============>.................] - ETA: 8s
28114944/62628765 [============>.................] - ETA: 8s
28311552/62628765 [============>.................] - ETA: 8s
28499968/62628765 [============>.................] - ETA: 8s
28696576/62628765 [============>.................] - ETA: 8s
28925952/62628765 [============>.................] - ETA: 8s
29138944/62628765 [============>.................] - ETA: 8s
29368320/62628765 [=============>................] - ETA: 8s
29630464/62628765 [=============>................] - ETA: 8s
29884416/62628765 [=============>................] - ETA: 8s
30113792/62628765 [=============>................] - ETA: 8s
30236672/62628765 [=============>................] - ETA: 8s
30392320/62628765 [=============>................] - ETA: 8s
30507008/62628765 [=============>................] - ETA: 8s
30646272/62628765 [=============>................] - ETA: 7s
30801920/62628765 [=============>................] - ETA: 7s
30965760/62628765 [=============>................] - ETA: 7s
31145984/62628765 [=============>................] - ETA: 7s
31334400/62628765 [==============>...............] - ETA: 7s
31514624/62628765 [==============>...............] - ETA: 7s
31719424/62628765 [==============>...............] - ETA: 7s
31932416/62628765 [==============>...............] - ETA: 7s
32137216/62628765 [==============>...............] - ETA: 7s
32374784/62628765 [==============>...............] - ETA: 7s
32636928/62628765 [==============>...............] - ETA: 7s
32907264/62628765 [==============>...............] - ETA: 7s
33169408/62628765 [==============>...............] - ETA: 7s
33431552/62628765 [===============>..............] - ETA: 7s
33726464/62628765 [===============>..............] - ETA: 7s
33988608/62628765 [===============>..............] - ETA: 7s
34299904/62628765 [===============>..............] - ETA: 7s
34603008/62628765 [===============>..............] - ETA: 6s
34824192/62628765 [===============>..............] - ETA: 6s
35020800/62628765 [===============>..............] - ETA: 6s
35168256/62628765 [===============>..............] - ETA: 6s
35332096/62628765 [===============>..............] - ETA: 6s
35504128/62628765 [================>.............] - ETA: 6s
35667968/62628765 [================>.............] - ETA: 6s
35708928/62628765 [================>.............] - ETA: 6s
35864576/62628765 [================>.............] - ETA: 6s
35946496/62628765 [================>.............] - ETA: 6s
36036608/62628765 [================>.............] - ETA: 6s
36159488/62628765 [================>.............] - ETA: 6s
36274176/62628765 [================>.............] - ETA: 6s
36413440/62628765 [================>.............] - ETA: 6s
36569088/62628765 [================>.............] - ETA: 6s
36732928/62628765 [================>.............] - ETA: 6s
36904960/62628765 [================>.............] - ETA: 6s
37076992/62628765 [================>.............] - ETA: 6s
37249024/62628765 [================>.............] - ETA: 6s
37445632/62628765 [================>.............] - ETA: 6s
37650432/62628765 [=================>............] - ETA: 6s
37855232/62628765 [=================>............] - ETA: 6s
38076416/62628765 [=================>............] - ETA: 6s
38297600/62628765 [=================>............] - ETA: 6s
38526976/62628765 [=================>............] - ETA: 6s
38780928/62628765 [=================>............] - ETA: 6s
39034880/62628765 [=================>............] - ETA: 5s
39297024/62628765 [=================>............] - ETA: 5s
39444480/62628765 [=================>............] - ETA: 5s
39641088/62628765 [=================>............] - ETA: 5s
39780352/62628765 [==================>...........] - ETA: 5s
39927808/62628765 [==================>...........] - ETA: 5s
40091648/62628765 [==================>...........] - ETA: 5s
40206336/62628765 [==================>...........] - ETA: 5s
40304640/62628765 [==================>...........] - ETA: 5s
40386560/62628765 [==================>...........] - ETA: 5s
40476672/62628765 [==================>...........] - ETA: 5s
40591360/62628765 [==================>...........] - ETA: 5s
40706048/62628765 [==================>...........] - ETA: 5s
40853504/62628765 [==================>...........] - ETA: 5s
41000960/62628765 [==================>...........] - ETA: 5s
41132032/62628765 [==================>...........] - ETA: 5s
41222144/62628765 [==================>...........] - ETA: 5s
41295872/62628765 [==================>...........] - ETA: 5s
41385984/62628765 [==================>...........] - ETA: 5s
41467904/62628765 [==================>...........] - ETA: 5s
41582592/62628765 [==================>...........] - ETA: 5s
41705472/62628765 [==================>...........] - ETA: 5s
41828352/62628765 [===================>..........] - ETA: 5s
41984000/62628765 [===================>..........] - ETA: 5s
42139648/62628765 [===================>..........] - ETA: 5s
42303488/62628765 [===================>..........] - ETA: 5s
42475520/62628765 [===================>..........] - ETA: 5s
42672128/62628765 [===================>..........] - ETA: 5s
42852352/62628765 [===================>..........] - ETA: 5s
43040768/62628765 [===================>..........] - ETA: 5s
43220992/62628765 [===================>..........] - ETA: 5s
43442176/62628765 [===================>..........] - ETA: 5s
43638784/62628765 [===================>..........] - ETA: 5s
43892736/62628765 [====================>.........] - ETA: 5s
44154880/62628765 [====================>.........] - ETA: 4s
44400640/62628765 [====================>.........] - ETA: 4s
44646400/62628765 [====================>.........] - ETA: 4s
44941312/62628765 [====================>.........] - ETA: 4s
45219840/62628765 [====================>.........] - ETA: 4s
45481984/62628765 [====================>.........] - ETA: 4s
45793280/62628765 [====================>.........] - ETA: 4s
46088192/62628765 [=====================>........] - ETA: 4s
46407680/62628765 [=====================>........] - ETA: 4s
46743552/62628765 [=====================>........] - ETA: 4s
47054848/62628765 [=====================>........] - ETA: 4s
47407104/62628765 [=====================>........] - ETA: 3s
47587328/62628765 [=====================>........] - ETA: 3s
47808512/62628765 [=====================>........] - ETA: 3s
47931392/62628765 [=====================>........] - ETA: 3s
48021504/62628765 [======================>.......] - ETA: 3s
48070656/62628765 [======================>.......] - ETA: 3s
48136192/62628765 [======================>.......] - ETA: 3s
48201728/62628765 [======================>.......] - ETA: 3s
48267264/62628765 [======================>.......] - ETA: 3s
48324608/62628765 [======================>.......] - ETA: 3s
48381952/62628765 [======================>.......] - ETA: 3s
48439296/62628765 [======================>.......] - ETA: 3s
48513024/62628765 [======================>.......] - ETA: 3s
48594944/62628765 [======================>.......] - ETA: 3s
48693248/62628765 [======================>.......] - ETA: 3s
48750592/62628765 [======================>.......] - ETA: 3s
48816128/62628765 [======================>.......] - ETA: 3s
48873472/62628765 [======================>.......] - ETA: 3s
48930816/62628765 [======================>.......] - ETA: 3s
49004544/62628765 [======================>.......] - ETA: 3s
49086464/62628765 [======================>.......] - ETA: 3s
49176576/62628765 [======================>.......] - ETA: 3s
49283072/62628765 [======================>.......] - ETA: 3s
49405952/62628765 [======================>.......] - ETA: 3s
49537024/62628765 [======================>.......] - ETA: 3s
49676288/62628765 [======================>.......] - ETA: 3s
49815552/62628765 [======================>.......] - ETA: 3s
49963008/62628765 [======================>.......] - ETA: 3s
50135040/62628765 [=======================>......] - ETA: 3s
50323456/62628765 [=======================>......] - ETA: 3s
50520064/62628765 [=======================>......] - ETA: 3s
50724864/62628765 [=======================>......] - ETA: 3s
50905088/62628765 [=======================>......] - ETA: 3s
51134464/62628765 [=======================>......] - ETA: 3s
51339264/62628765 [=======================>......] - ETA: 3s
51576832/62628765 [=======================>......] - ETA: 3s
51798016/62628765 [=======================>......] - ETA: 2s
52068352/62628765 [=======================>......] - ETA: 2s
52355072/62628765 [========================>.....] - ETA: 2s
52633600/62628765 [========================>.....] - ETA: 2s
52936704/62628765 [========================>.....] - ETA: 2s
53248000/62628765 [========================>.....] - ETA: 2s
53534720/62628765 [========================>.....] - ETA: 2s
53846016/62628765 [========================>.....] - ETA: 2s
54165504/62628765 [========================>.....] - ETA: 2s
54501376/62628765 [=========================>....] - ETA: 2s
54837248/62628765 [=========================>....] - ETA: 2s
55189504/62628765 [=========================>....] - ETA: 2s
55541760/62628765 [=========================>....] - ETA: 1s
55730176/62628765 [=========================>....] - ETA: 1s
55934976/62628765 [=========================>....] - ETA: 1s
56098816/62628765 [=========================>....] - ETA: 1s
56287232/62628765 [=========================>....] - ETA: 1s
56483840/62628765 [==========================>...] - ETA: 1s
56664064/62628765 [==========================>...] - ETA: 1s
56803328/62628765 [==========================>...] - ETA: 1s
56909824/62628765 [==========================>...] - ETA: 1s
56999936/62628765 [==========================>...] - ETA: 1s
57131008/62628765 [==========================>...] - ETA: 1s
57270272/62628765 [==========================>...] - ETA: 1s
57417728/62628765 [==========================>...] - ETA: 1s
57581568/62628765 [==========================>...] - ETA: 1s
57753600/62628765 [==========================>...] - ETA: 1s
57942016/62628765 [==========================>...] - ETA: 1s
58138624/62628765 [==========================>...] - ETA: 1s
58343424/62628765 [==========================>...] - ETA: 1s
58564608/62628765 [===========================>..] - ETA: 1s
58810368/62628765 [===========================>..] - ETA: 1s
59047936/62628765 [===========================>..] - ETA: 0s
59285504/62628765 [===========================>..] - ETA: 0s
59375616/62628765 [===========================>..] - ETA: 0s
59523072/62628765 [===========================>..] - ETA: 0s
59654144/62628765 [===========================>..] - ETA: 0s
59793408/62628765 [===========================>..] - ETA: 0s
59850752/62628765 [===========================>..] - ETA: 0s
59932672/62628765 [===========================>..] - ETA: 0s
59973632/62628765 [===========================>..] - ETA: 0s
60022784/62628765 [===========================>..] - ETA: 0s
60080128/62628765 [===========================>..] - ETA: 0s
60145664/62628765 [===========================>..] - ETA: 0s
60227584/62628765 [===========================>..] - ETA: 0s
60317696/62628765 [===========================>..] - ETA: 0s
60424192/62628765 [===========================>..] - ETA: 0s
60538880/62628765 [===========================>..] - ETA: 0s
60661760/62628765 [============================>.] - ETA: 0s
60784640/62628765 [============================>.] - ETA: 0s
60923904/62628765 [============================>.] - ETA: 0s
61087744/62628765 [============================>.] - ETA: 0s
61251584/62628765 [============================>.] - ETA: 0s
61440000/62628765 [============================>.] - ETA: 0s
61636608/62628765 [============================>.] - ETA: 0s
61718528/62628765 [============================>.] - ETA: 0s
61833216/62628765 [============================>.] - ETA: 0s
61915136/62628765 [============================>.] - ETA: 0s
62029824/62628765 [============================>.] - ETA: 0s
62152704/62628765 [============================>.] - ETA: 0s
62275584/62628765 [============================>.] - ETA: 0s
62406656/62628765 [============================>.] - ETA: 0s
62488576/62628765 [============================>.] - ETA: 0s
62586880/62628765 [============================>.] - ETA: 0s
62628765/62628765 [==============================] - 18s 0us/step
Wanted words and labels:
 {'six': 23, 'three': 25, 'seven': 21, 'bed': 1, 'eight': 6, 'yes': 31, 'cat': 3, 'on': 18, 'one': 19, 'stop': 24, 'two': 27, 'house': 11, 'five': 7, 'down': 5, 'four': 8, 'go': 9, 'up': 28, 'learn': 12, 'no': 16, 'bird': 2, 'zero': 32, 'nine': 15, 'visual': 29, 'wow': 30, 'sheila': 22, 'marvin': 14, 'off': 17, 'right': 20, 'left': 13, 'happy': 10, 'dog': 4, 'tree': 26, '_silence_': 0}

2. Load a pre-trained native Keras model

The model consists of:

  • a first convolutional layer accepting dense inputs (images),

  • several separable convolutional layers preserving spatial dimensions,

  • a global pooling reducing the spatial dimensions to a single pixel,

  • a last separable convolutional to reduce the number of outputs

  • a final fully connected layer to classify words

All layers are followed by a batch normalization and a ReLU activation.

This model was obtained with unconstrained float weights and activations after 16 epochs of training.

from tensorflow.keras.models import load_model

# Retrieve the model file from the BrainChip data server
model_file = get_file("ds_cnn_kws.h5",
                      "http://data.brainchip.com/models/ds_cnn/ds_cnn_kws.h5",
                      cache_subdir='models')

# Load the native Keras pre-trained model
model_keras = load_model(model_file)
model_keras.summary()

Downloading data from http://data.brainchip.com/models/ds_cnn/ds_cnn_kws.h5

  8192/162912 [>.............................] - ETA: 0s
 90112/162912 [===============>..............] - ETA: 0s
162912/162912 [==============================] - 0s 1us/step
Model: "ds_cnn_kws"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 input (InputLayer)          [(None, 49, 10, 1)]       0

 rescaling (Rescaling)       (None, 49, 10, 1)         0

 conv_0 (Conv2D)             (None, 25, 5, 64)         1600

 conv_0/BN (BatchNormalizati  (None, 25, 5, 64)        256
 on)

 conv_0/relu (ReLU)          (None, 25, 5, 64)         0

 separable_1 (SeparableConv2  (None, 25, 5, 64)        4672
 D)

 separable_1/BN (BatchNormal  (None, 25, 5, 64)        256
 ization)

 separable_1/relu (ReLU)     (None, 25, 5, 64)         0

 separable_2 (SeparableConv2  (None, 25, 5, 64)        4672
 D)

 separable_2/BN (BatchNormal  (None, 25, 5, 64)        256
 ization)

 separable_2/relu (ReLU)     (None, 25, 5, 64)         0

 separable_3 (SeparableConv2  (None, 25, 5, 64)        4672
 D)

 separable_3/BN (BatchNormal  (None, 25, 5, 64)        256
 ization)

 separable_3/relu (ReLU)     (None, 25, 5, 64)         0

 separable_4 (SeparableConv2  (None, 25, 5, 64)        4672
 D)

 separable_4/global_avg (Glo  (None, 64)               0
 balAveragePooling2D)

 separable_4/BN (BatchNormal  (None, 64)               256
 ization)

 separable_4/relu (ReLU)     (None, 64)                0

 reshape_1 (Reshape)         (None, 1, 1, 64)          0

 flatten (Flatten)           (None, 64)                0

 dense_5 (Dense)             (None, 33)                2145

 act_softmax (Activation)    (None, 33)                0

=================================================================
Total params: 23,713
Trainable params: 23,073
Non-trainable params: 640
_________________________________________________________________

import numpy as np

from sklearn.metrics import accuracy_score

# Check Keras Model performance
potentials_keras = model_keras.predict(x_valid)
preds_keras = np.squeeze(np.argmax(potentials_keras, 1))

accuracy = accuracy_score(y_valid, preds_keras)
print("Accuracy: " + "{0:.2f}".format(100 * accuracy) + "%")

  1/308 [..............................] - ETA: 32s
 47/308 [===>..........................] - ETA: 0s 
 94/308 [========>.....................] - ETA: 0s
141/308 [============>.................] - ETA: 0s
188/308 [=================>............] - ETA: 0s
233/308 [=====================>........] - ETA: 0s
279/308 [==========================>...] - ETA: 0s
308/308 [==============================] - 0s 1ms/step
Accuracy: 92.85%

3. Load a pre-trained quantized Keras model satisfying Akida NSoC requirements

The above native Keras model is quantized and fine-tuned to get a quantized Keras model satisfying the Akida NSoC requirements. The first convolutional layer uses 8 bits weights, but other layers use 4 bits weights.

All activations are 4 bits except for the final Separable Convolutional that uses binary activations.

Pre-trained weights were obtained after a few training episodes:

  • we train the model with quantized activations only, with weights initialized from those trained in the previous episode (native Keras model),

  • then, we train the model with quantized weights, with both weights and activations initialized from those trained in the previous episode,

  • finally, we train the model with quantized weights and activations and by gradually increasing quantization in the last layer.

The table below summarizes the results obtained when preparing the weights stored under http://data.brainchip.com/models/ds_cnn/ :

Episode

Weights Quant.

Activ. Quant. / last layer

Accuracy

Epochs

1

N/A

N/A

93.06 %

16

2

N/A

4 bits / 4 bits

92.30 %

16

3

8/4 bits

4 bits / 4 bits

92.11 %

16

4

8/4 bits

4 bits / 3 bits

92.38 %

16

5

8/4 bits

4 bits / 2 bits

92.23 %

16

6

8/4 bits

4 bits / 1 bit

92.22 %

16

 
from akida_models import ds_cnn_kws_pretrained

# Load the pre-trained quantized model
model_keras_quantized = ds_cnn_kws_pretrained()
model_keras_quantized.summary()

# Check Model performance
potentials_keras_q = model_keras_quantized.predict(x_valid)
preds_keras_q = np.squeeze(np.argmax(potentials_keras_q, 1))

accuracy_q = accuracy_score(y_valid, preds_keras_q)
print("Accuracy: " + "{0:.2f}".format(100 * accuracy_q) + "%")

Downloading data from http://data.brainchip.com/models/ds_cnn/ds_cnn_kws_iq8_wq4_aq4_laq1.h5.

     0/134616 [..............................] - ETA: 0s
 90112/134616 [===================>..........] - ETA: 0s
134616/134616 [==============================] - 0s 0us/step
Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling (Rescaling)       (None, 49, 10, 1)         0

 conv_0 (QuantizedConv2D)    (None, 25, 5, 64)         1664

 conv_0/relu (QuantizedReLU)  (None, 25, 5, 64)        0

 separable_1 (QuantizedSepar  (None, 25, 5, 64)        4736
 ableConv2D)

 separable_1/relu (Quantized  (None, 25, 5, 64)        0
 ReLU)

 separable_2 (QuantizedSepar  (None, 25, 5, 64)        4736
 ableConv2D)

 separable_2/relu (Quantized  (None, 25, 5, 64)        0
 ReLU)

 separable_3 (QuantizedSepar  (None, 25, 5, 64)        4736
 ableConv2D)

 separable_3/relu (Quantized  (None, 25, 5, 64)        0
 ReLU)

 separable_4 (QuantizedSepar  (None, 25, 5, 64)        4736
 ableConv2D)

 separable_4/global_avg (Glo  (None, 64)               0
 balAveragePooling2D)

 separable_4/relu (Quantized  (None, 64)               0
 ReLU)

 reshape_1 (Reshape)         (None, 1, 1, 64)          0

 flatten (Flatten)           (None, 64)                0

 dense_5 (QuantizedDense)    (None, 33)                2145

 act_softmax (Activation)    (None, 33)                0

=================================================================
Total params: 22,753
Trainable params: 22,753
Non-trainable params: 0
_________________________________________________________________

  1/308 [..............................] - ETA: 1:14
 34/308 [==>...........................] - ETA: 0s  
 68/308 [=====>........................] - ETA: 0s
101/308 [========>.....................] - ETA: 0s
135/308 [============>.................] - ETA: 0s
169/308 [===============>..............] - ETA: 0s
203/308 [==================>...........] - ETA: 0s
237/308 [======================>.......] - ETA: 0s
270/308 [=========================>....] - ETA: 0s
302/308 [============================>.] - ETA: 0s
308/308 [==============================] - 1s 2ms/step
Accuracy: 91.91%

4. Conversion to Akida

We convert the model to Akida and then evaluate the performances on the dataset.

from cnn2snn import convert

# Convert the model
model_akida = convert(model_keras_quantized)
model_akida.summary()

                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[49, 10, 1]  [1, 1, 33]    1          6
______________________________________________

_______________________________________________________
Layer (type)             Output shape  Kernel shape

============= SW/conv_0-dense_5 (Software) ============

conv_0 (InputConv.)      [25, 5, 64]   (5, 5, 1, 64)
_______________________________________________________
separable_1 (Sep.Conv.)  [25, 5, 64]   (3, 3, 64, 1)
_______________________________________________________
                                       (1, 1, 64, 64)
_______________________________________________________
separable_2 (Sep.Conv.)  [25, 5, 64]   (3, 3, 64, 1)
_______________________________________________________
                                       (1, 1, 64, 64)
_______________________________________________________
separable_3 (Sep.Conv.)  [25, 5, 64]   (3, 3, 64, 1)
_______________________________________________________
                                       (1, 1, 64, 64)
_______________________________________________________
separable_4 (Sep.Conv.)  [1, 1, 64]    (3, 3, 64, 1)
_______________________________________________________
                                       (1, 1, 64, 64)
_______________________________________________________
dense_5 (Fully.)         [1, 1, 33]    (1, 1, 64, 33)
_______________________________________________________

# Check Akida model performance
preds_akida = model_akida.predict_classes(x_valid, num_classes=num_classes)

accuracy = accuracy_score(y_valid, preds_akida)
print("Accuracy: " + "{0:.2f}".format(100 * accuracy) + "%")

# For non-regression purpose
assert accuracy > 0.9

Accuracy: 91.73%

5. Confusion matrix

The confusion matrix provides a good summary of what mistakes the network is making.

Per scikit-learn convention it displays the true class in each row (ie on each row you can see what the network predicted for the corresponding word).

Please refer to the Tensorflow audio recognition example for a detailed explanation of the confusion matrix.

import itertools
import matplotlib.pyplot as plt

from sklearn.metrics import confusion_matrix

# Create confusion matrix
cm = confusion_matrix(y_valid, preds_akida,
                      labels=list(word_to_index.values()))

# Normalize
cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

# Display confusion matrix
plt.rcParams["figure.figsize"] = (16, 16)
plt.figure()

title = 'Confusion matrix'
cmap = plt.cm.Blues

plt.imshow(cm, interpolation='nearest', cmap=cmap)
plt.title(title)
plt.colorbar()
tick_marks = np.arange(len(word_to_index))
plt.xticks(tick_marks, word_to_index, rotation=45)
plt.yticks(tick_marks, word_to_index)

thresh = cm.max() / 2.
for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
    plt.text(j,
             i,
             format(cm[i, j], '.2f'),
             horizontalalignment="center",
             color="white" if cm[i, j] > thresh else "black")

plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.autoscale()
plt.show()
Confusion matrix

Total running time of the script: ( 0 minutes 24.795 seconds)

Gallery generated by Sphinx-Gallery