Note
Click here to download the full example code
CNN conversion flow tutorial
This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.
The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.
Note
Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.
MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.
1. Load and reshape MNIST dataset
After loading, we make 2 transformations on the dataset:
Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.
Note
At this point, we’ll set aside the raw data for testing our converted model in the Akida runtime later.
Rescale the 8-bit loaded data to the range 0-to-1 for training.
Note
Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.
This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.
Also note that we store the scaling values input_scaling for
use when preparing the model for the Akida neuromorphic IP. The
implementation of the Akida neural network allows us to completely
skip the rescaling step (i.e. the Akida model should be fed with
the raw 8-bit values) but that does require information about what
scaling was applied prior to training - see below for more details.
from tensorflow import keras
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)
# Set aside raw test data for use with Akida runtime later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test
# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a
2. Model definition
Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.
An appropriate model for MNIST (inspired by this example) might look something like the following:
model_keras = keras.models.Sequential([
keras.layers.Conv2D(
filters=32, kernel_size=3, input_shape=(28, 28, 1), strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.SeparableConv2D(
filters=64, kernel_size=3, padding='same', strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 13, 13, 32) 320
batch_normalization (BatchN (None, 13, 13, 32) 128
ormalization)
re_lu (ReLU) (None, 13, 13, 32) 0
separable_conv2d (Separable (None, 7, 7, 64) 2400
Conv2D)
batch_normalization_1 (Batc (None, 7, 7, 64) 256
hNormalization)
re_lu_1 (ReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________
The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):
Standard Conv2D and Dense layers are supported
Hidden layers must be followed by a ReLU layer.
BatchNormalization must always happen before activations.
Convolutional blocks can optionally be followed by a MaxPooling.
The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).
from cnn2snn import check_model_compatibility
print("Model compatible for Akida conversion:",
check_model_compatibility(model_keras))
Model compatible for Akida conversion: True
3. Model training
Before going any further, train the model and get its performance. The created model should have achieved a test accuracy over 98% after 10 epochs.
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Epoch 1/10
1/1688 [..............................] - ETA: 12:50 - loss: 2.5593 - accuracy: 0.1562
28/1688 [..............................] - ETA: 3s - loss: 1.4299 - accuracy: 0.5301
57/1688 [>.............................] - ETA: 2s - loss: 1.0237 - accuracy: 0.6623
85/1688 [>.............................] - ETA: 2s - loss: 0.8121 - accuracy: 0.7390
112/1688 [>.............................] - ETA: 2s - loss: 0.7061 - accuracy: 0.7729
140/1688 [=>............................] - ETA: 2s - loss: 0.6344 - accuracy: 0.7973
169/1688 [==>...........................] - ETA: 2s - loss: 0.5764 - accuracy: 0.8173
197/1688 [==>...........................] - ETA: 2s - loss: 0.5367 - accuracy: 0.8311
225/1688 [==>...........................] - ETA: 2s - loss: 0.4994 - accuracy: 0.8431
253/1688 [===>..........................] - ETA: 2s - loss: 0.4725 - accuracy: 0.8531
281/1688 [===>..........................] - ETA: 2s - loss: 0.4452 - accuracy: 0.8621
309/1688 [====>.........................] - ETA: 2s - loss: 0.4217 - accuracy: 0.8695
337/1688 [====>.........................] - ETA: 2s - loss: 0.4051 - accuracy: 0.8743
365/1688 [=====>........................] - ETA: 2s - loss: 0.3876 - accuracy: 0.8801
393/1688 [=====>........................] - ETA: 2s - loss: 0.3713 - accuracy: 0.8861
421/1688 [======>.......................] - ETA: 2s - loss: 0.3561 - accuracy: 0.8907
449/1688 [======>.......................] - ETA: 2s - loss: 0.3424 - accuracy: 0.8949
477/1688 [=======>......................] - ETA: 2s - loss: 0.3315 - accuracy: 0.8979
505/1688 [=======>......................] - ETA: 2s - loss: 0.3209 - accuracy: 0.9014
532/1688 [========>.....................] - ETA: 2s - loss: 0.3135 - accuracy: 0.9037
561/1688 [========>.....................] - ETA: 2s - loss: 0.3056 - accuracy: 0.9061
590/1688 [=========>....................] - ETA: 2s - loss: 0.2988 - accuracy: 0.9083
618/1688 [=========>....................] - ETA: 1s - loss: 0.2907 - accuracy: 0.9108
646/1688 [==========>...................] - ETA: 1s - loss: 0.2839 - accuracy: 0.9129
674/1688 [==========>...................] - ETA: 1s - loss: 0.2761 - accuracy: 0.9153
702/1688 [===========>..................] - ETA: 1s - loss: 0.2695 - accuracy: 0.9173
730/1688 [===========>..................] - ETA: 1s - loss: 0.2643 - accuracy: 0.9190
758/1688 [============>.................] - ETA: 1s - loss: 0.2577 - accuracy: 0.9212
786/1688 [============>.................] - ETA: 1s - loss: 0.2537 - accuracy: 0.9224
814/1688 [=============>................] - ETA: 1s - loss: 0.2507 - accuracy: 0.9235
842/1688 [=============>................] - ETA: 1s - loss: 0.2468 - accuracy: 0.9248
870/1688 [==============>...............] - ETA: 1s - loss: 0.2421 - accuracy: 0.9263
898/1688 [==============>...............] - ETA: 1s - loss: 0.2382 - accuracy: 0.9273
926/1688 [===============>..............] - ETA: 1s - loss: 0.2358 - accuracy: 0.9281
954/1688 [===============>..............] - ETA: 1s - loss: 0.2319 - accuracy: 0.9294
983/1688 [================>.............] - ETA: 1s - loss: 0.2286 - accuracy: 0.9304
1011/1688 [================>.............] - ETA: 1s - loss: 0.2254 - accuracy: 0.9313
1040/1688 [=================>............] - ETA: 1s - loss: 0.2217 - accuracy: 0.9324
1069/1688 [=================>............] - ETA: 1s - loss: 0.2185 - accuracy: 0.9333
1097/1688 [==================>...........] - ETA: 1s - loss: 0.2157 - accuracy: 0.9343
1125/1688 [==================>...........] - ETA: 1s - loss: 0.2130 - accuracy: 0.9350
1153/1688 [===================>..........] - ETA: 0s - loss: 0.2103 - accuracy: 0.9357
1181/1688 [===================>..........] - ETA: 0s - loss: 0.2076 - accuracy: 0.9364
1209/1688 [====================>.........] - ETA: 0s - loss: 0.2053 - accuracy: 0.9371
1237/1688 [====================>.........] - ETA: 0s - loss: 0.2028 - accuracy: 0.9379
1265/1688 [=====================>........] - ETA: 0s - loss: 0.2004 - accuracy: 0.9387
1293/1688 [=====================>........] - ETA: 0s - loss: 0.1974 - accuracy: 0.9397
1321/1688 [======================>.......] - ETA: 0s - loss: 0.1954 - accuracy: 0.9403
1348/1688 [======================>.......] - ETA: 0s - loss: 0.1940 - accuracy: 0.9407
1376/1688 [=======================>......] - ETA: 0s - loss: 0.1920 - accuracy: 0.9413
1403/1688 [=======================>......] - ETA: 0s - loss: 0.1907 - accuracy: 0.9418
1431/1688 [========================>.....] - ETA: 0s - loss: 0.1895 - accuracy: 0.9423
1459/1688 [========================>.....] - ETA: 0s - loss: 0.1883 - accuracy: 0.9428
1488/1688 [=========================>....] - ETA: 0s - loss: 0.1868 - accuracy: 0.9433
1516/1688 [=========================>....] - ETA: 0s - loss: 0.1851 - accuracy: 0.9438
1544/1688 [==========================>...] - ETA: 0s - loss: 0.1833 - accuracy: 0.9443
1571/1688 [==========================>...] - ETA: 0s - loss: 0.1815 - accuracy: 0.9448
1598/1688 [===========================>..] - ETA: 0s - loss: 0.1801 - accuracy: 0.9452
1626/1688 [===========================>..] - ETA: 0s - loss: 0.1781 - accuracy: 0.9458
1654/1688 [============================>.] - ETA: 0s - loss: 0.1766 - accuracy: 0.9462
1683/1688 [============================>.] - ETA: 0s - loss: 0.1749 - accuracy: 0.9467
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1747 - accuracy: 0.9468 - val_loss: 0.0684 - val_accuracy: 0.9820
Epoch 2/10
1/1688 [..............................] - ETA: 3s - loss: 0.0267 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0886 - accuracy: 0.9688
58/1688 [>.............................] - ETA: 2s - loss: 0.0720 - accuracy: 0.9774
86/1688 [>.............................] - ETA: 2s - loss: 0.0791 - accuracy: 0.9764
115/1688 [=>............................] - ETA: 2s - loss: 0.0751 - accuracy: 0.9780
143/1688 [=>............................] - ETA: 2s - loss: 0.0730 - accuracy: 0.9790
171/1688 [==>...........................] - ETA: 2s - loss: 0.0686 - accuracy: 0.9801
199/1688 [==>...........................] - ETA: 2s - loss: 0.0669 - accuracy: 0.9807
226/1688 [===>..........................] - ETA: 2s - loss: 0.0675 - accuracy: 0.9806
254/1688 [===>..........................] - ETA: 2s - loss: 0.0696 - accuracy: 0.9799
281/1688 [===>..........................] - ETA: 2s - loss: 0.0732 - accuracy: 0.9784
309/1688 [====>.........................] - ETA: 2s - loss: 0.0728 - accuracy: 0.9782
336/1688 [====>.........................] - ETA: 2s - loss: 0.0719 - accuracy: 0.9788
364/1688 [=====>........................] - ETA: 2s - loss: 0.0715 - accuracy: 0.9788
393/1688 [=====>........................] - ETA: 2s - loss: 0.0709 - accuracy: 0.9792
421/1688 [======>.......................] - ETA: 2s - loss: 0.0706 - accuracy: 0.9794
449/1688 [======>.......................] - ETA: 2s - loss: 0.0695 - accuracy: 0.9799
477/1688 [=======>......................] - ETA: 2s - loss: 0.0697 - accuracy: 0.9798
504/1688 [=======>......................] - ETA: 2s - loss: 0.0704 - accuracy: 0.9796
532/1688 [========>.....................] - ETA: 2s - loss: 0.0695 - accuracy: 0.9799
560/1688 [========>.....................] - ETA: 2s - loss: 0.0688 - accuracy: 0.9798
588/1688 [=========>....................] - ETA: 2s - loss: 0.0696 - accuracy: 0.9797
616/1688 [=========>....................] - ETA: 1s - loss: 0.0692 - accuracy: 0.9798
644/1688 [==========>...................] - ETA: 1s - loss: 0.0679 - accuracy: 0.9800
672/1688 [==========>...................] - ETA: 1s - loss: 0.0680 - accuracy: 0.9799
700/1688 [===========>..................] - ETA: 1s - loss: 0.0682 - accuracy: 0.9796
728/1688 [===========>..................] - ETA: 1s - loss: 0.0690 - accuracy: 0.9792
755/1688 [============>.................] - ETA: 1s - loss: 0.0684 - accuracy: 0.9792
783/1688 [============>.................] - ETA: 1s - loss: 0.0675 - accuracy: 0.9793
811/1688 [=============>................] - ETA: 1s - loss: 0.0675 - accuracy: 0.9794
840/1688 [=============>................] - ETA: 1s - loss: 0.0676 - accuracy: 0.9794
868/1688 [==============>...............] - ETA: 1s - loss: 0.0676 - accuracy: 0.9794
896/1688 [==============>...............] - ETA: 1s - loss: 0.0684 - accuracy: 0.9792
924/1688 [===============>..............] - ETA: 1s - loss: 0.0698 - accuracy: 0.9789
952/1688 [===============>..............] - ETA: 1s - loss: 0.0705 - accuracy: 0.9787
981/1688 [================>.............] - ETA: 1s - loss: 0.0707 - accuracy: 0.9788
1009/1688 [================>.............] - ETA: 1s - loss: 0.0709 - accuracy: 0.9787
1037/1688 [=================>............] - ETA: 1s - loss: 0.0712 - accuracy: 0.9784
1066/1688 [=================>............] - ETA: 1s - loss: 0.0710 - accuracy: 0.9785
1094/1688 [==================>...........] - ETA: 1s - loss: 0.0705 - accuracy: 0.9786
1122/1688 [==================>...........] - ETA: 1s - loss: 0.0707 - accuracy: 0.9785
1150/1688 [===================>..........] - ETA: 0s - loss: 0.0701 - accuracy: 0.9786
1178/1688 [===================>..........] - ETA: 0s - loss: 0.0704 - accuracy: 0.9787
1205/1688 [====================>.........] - ETA: 0s - loss: 0.0710 - accuracy: 0.9784
1233/1688 [====================>.........] - ETA: 0s - loss: 0.0708 - accuracy: 0.9785
1260/1688 [=====================>........] - ETA: 0s - loss: 0.0707 - accuracy: 0.9784
1288/1688 [=====================>........] - ETA: 0s - loss: 0.0705 - accuracy: 0.9785
1315/1688 [======================>.......] - ETA: 0s - loss: 0.0707 - accuracy: 0.9785
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0704 - accuracy: 0.9785
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0705 - accuracy: 0.9784
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0707 - accuracy: 0.9784
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0703 - accuracy: 0.9785
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0702 - accuracy: 0.9786
1481/1688 [=========================>....] - ETA: 0s - loss: 0.0699 - accuracy: 0.9786
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0696 - accuracy: 0.9787
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0699 - accuracy: 0.9787
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0697 - accuracy: 0.9787
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0695 - accuracy: 0.9787
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0693 - accuracy: 0.9788
1649/1688 [============================>.] - ETA: 0s - loss: 0.0693 - accuracy: 0.9787
1677/1688 [============================>.] - ETA: 0s - loss: 0.0692 - accuracy: 0.9788
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0691 - accuracy: 0.9788 - val_loss: 0.0643 - val_accuracy: 0.9842
Epoch 3/10
1/1688 [..............................] - ETA: 3s - loss: 0.0097 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0502 - accuracy: 0.9833
57/1688 [>.............................] - ETA: 2s - loss: 0.0365 - accuracy: 0.9890
84/1688 [>.............................] - ETA: 2s - loss: 0.0431 - accuracy: 0.9877
112/1688 [>.............................] - ETA: 2s - loss: 0.0384 - accuracy: 0.9883
140/1688 [=>............................] - ETA: 2s - loss: 0.0365 - accuracy: 0.9891
168/1688 [=>............................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9888
196/1688 [==>...........................] - ETA: 2s - loss: 0.0372 - accuracy: 0.9890
223/1688 [==>...........................] - ETA: 2s - loss: 0.0370 - accuracy: 0.9891
251/1688 [===>..........................] - ETA: 2s - loss: 0.0383 - accuracy: 0.9887
279/1688 [===>..........................] - ETA: 2s - loss: 0.0405 - accuracy: 0.9881
308/1688 [====>.........................] - ETA: 2s - loss: 0.0434 - accuracy: 0.9878
336/1688 [====>.........................] - ETA: 2s - loss: 0.0442 - accuracy: 0.9877
364/1688 [=====>........................] - ETA: 2s - loss: 0.0443 - accuracy: 0.9876
392/1688 [=====>........................] - ETA: 2s - loss: 0.0457 - accuracy: 0.9872
420/1688 [======>.......................] - ETA: 2s - loss: 0.0460 - accuracy: 0.9869
448/1688 [======>.......................] - ETA: 2s - loss: 0.0461 - accuracy: 0.9867
476/1688 [=======>......................] - ETA: 2s - loss: 0.0451 - accuracy: 0.9869
504/1688 [=======>......................] - ETA: 2s - loss: 0.0464 - accuracy: 0.9864
532/1688 [========>.....................] - ETA: 2s - loss: 0.0468 - accuracy: 0.9861
560/1688 [========>.....................] - ETA: 2s - loss: 0.0473 - accuracy: 0.9860
588/1688 [=========>....................] - ETA: 2s - loss: 0.0483 - accuracy: 0.9859
616/1688 [=========>....................] - ETA: 1s - loss: 0.0491 - accuracy: 0.9855
644/1688 [==========>...................] - ETA: 1s - loss: 0.0486 - accuracy: 0.9856
672/1688 [==========>...................] - ETA: 1s - loss: 0.0498 - accuracy: 0.9852
700/1688 [===========>..................] - ETA: 1s - loss: 0.0494 - accuracy: 0.9853
728/1688 [===========>..................] - ETA: 1s - loss: 0.0496 - accuracy: 0.9852
756/1688 [============>.................] - ETA: 1s - loss: 0.0495 - accuracy: 0.9852
785/1688 [============>.................] - ETA: 1s - loss: 0.0503 - accuracy: 0.9847
813/1688 [=============>................] - ETA: 1s - loss: 0.0510 - accuracy: 0.9845
840/1688 [=============>................] - ETA: 1s - loss: 0.0513 - accuracy: 0.9845
868/1688 [==============>...............] - ETA: 1s - loss: 0.0505 - accuracy: 0.9848
896/1688 [==============>...............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9849
924/1688 [===============>..............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9848
952/1688 [===============>..............] - ETA: 1s - loss: 0.0497 - accuracy: 0.9850
979/1688 [================>.............] - ETA: 1s - loss: 0.0500 - accuracy: 0.9849
1007/1688 [================>.............] - ETA: 1s - loss: 0.0493 - accuracy: 0.9850
1035/1688 [=================>............] - ETA: 1s - loss: 0.0498 - accuracy: 0.9848
1064/1688 [=================>............] - ETA: 1s - loss: 0.0498 - accuracy: 0.9847
1091/1688 [==================>...........] - ETA: 1s - loss: 0.0499 - accuracy: 0.9847
1119/1688 [==================>...........] - ETA: 1s - loss: 0.0498 - accuracy: 0.9847
1148/1688 [===================>..........] - ETA: 0s - loss: 0.0496 - accuracy: 0.9848
1176/1688 [===================>..........] - ETA: 0s - loss: 0.0498 - accuracy: 0.9847
1203/1688 [====================>.........] - ETA: 0s - loss: 0.0504 - accuracy: 0.9847
1231/1688 [====================>.........] - ETA: 0s - loss: 0.0498 - accuracy: 0.9848
1259/1688 [=====================>........] - ETA: 0s - loss: 0.0495 - accuracy: 0.9849
1287/1688 [=====================>........] - ETA: 0s - loss: 0.0498 - accuracy: 0.9847
1315/1688 [======================>.......] - ETA: 0s - loss: 0.0504 - accuracy: 0.9846
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0504 - accuracy: 0.9846
1371/1688 [=======================>......] - ETA: 0s - loss: 0.0505 - accuracy: 0.9845
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0502 - accuracy: 0.9847
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0502 - accuracy: 0.9847
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0502 - accuracy: 0.9846
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0504 - accuracy: 0.9845
1510/1688 [=========================>....] - ETA: 0s - loss: 0.0503 - accuracy: 0.9845
1538/1688 [==========================>...] - ETA: 0s - loss: 0.0513 - accuracy: 0.9844
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0514 - accuracy: 0.9842
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0518 - accuracy: 0.9841
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0517 - accuracy: 0.9841
1649/1688 [============================>.] - ETA: 0s - loss: 0.0519 - accuracy: 0.9840
1677/1688 [============================>.] - ETA: 0s - loss: 0.0521 - accuracy: 0.9838
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0522 - accuracy: 0.9838 - val_loss: 0.0726 - val_accuracy: 0.9822
Epoch 4/10
1/1688 [..............................] - ETA: 3s - loss: 0.0049 - accuracy: 1.0000
29/1688 [..............................] - ETA: 2s - loss: 0.0591 - accuracy: 0.9795
57/1688 [>.............................] - ETA: 2s - loss: 0.0496 - accuracy: 0.9819
84/1688 [>.............................] - ETA: 2s - loss: 0.0490 - accuracy: 0.9836
112/1688 [>.............................] - ETA: 2s - loss: 0.0439 - accuracy: 0.9858
140/1688 [=>............................] - ETA: 2s - loss: 0.0416 - accuracy: 0.9875
167/1688 [=>............................] - ETA: 2s - loss: 0.0399 - accuracy: 0.9875
196/1688 [==>...........................] - ETA: 2s - loss: 0.0383 - accuracy: 0.9880
223/1688 [==>...........................] - ETA: 2s - loss: 0.0384 - accuracy: 0.9884
250/1688 [===>..........................] - ETA: 2s - loss: 0.0383 - accuracy: 0.9881
277/1688 [===>..........................] - ETA: 2s - loss: 0.0371 - accuracy: 0.9883
305/1688 [====>.........................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9882
333/1688 [====>.........................] - ETA: 2s - loss: 0.0379 - accuracy: 0.9879
361/1688 [=====>........................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9877
390/1688 [=====>........................] - ETA: 2s - loss: 0.0366 - accuracy: 0.9880
418/1688 [======>.......................] - ETA: 2s - loss: 0.0364 - accuracy: 0.9879
446/1688 [======>.......................] - ETA: 2s - loss: 0.0362 - accuracy: 0.9881
474/1688 [=======>......................] - ETA: 2s - loss: 0.0363 - accuracy: 0.9878
502/1688 [=======>......................] - ETA: 2s - loss: 0.0368 - accuracy: 0.9876
530/1688 [========>.....................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9875
558/1688 [========>.....................] - ETA: 2s - loss: 0.0380 - accuracy: 0.9875
586/1688 [=========>....................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9876
614/1688 [=========>....................] - ETA: 1s - loss: 0.0380 - accuracy: 0.9876
642/1688 [==========>...................] - ETA: 1s - loss: 0.0399 - accuracy: 0.9871
670/1688 [==========>...................] - ETA: 1s - loss: 0.0397 - accuracy: 0.9872
698/1688 [===========>..................] - ETA: 1s - loss: 0.0396 - accuracy: 0.9872
726/1688 [===========>..................] - ETA: 1s - loss: 0.0398 - accuracy: 0.9871
754/1688 [============>.................] - ETA: 1s - loss: 0.0396 - accuracy: 0.9872
781/1688 [============>.................] - ETA: 1s - loss: 0.0397 - accuracy: 0.9870
809/1688 [=============>................] - ETA: 1s - loss: 0.0394 - accuracy: 0.9871
836/1688 [=============>................] - ETA: 1s - loss: 0.0391 - accuracy: 0.9873
864/1688 [==============>...............] - ETA: 1s - loss: 0.0396 - accuracy: 0.9871
893/1688 [==============>...............] - ETA: 1s - loss: 0.0405 - accuracy: 0.9869
921/1688 [===============>..............] - ETA: 1s - loss: 0.0410 - accuracy: 0.9866
949/1688 [===============>..............] - ETA: 1s - loss: 0.0408 - accuracy: 0.9868
976/1688 [================>.............] - ETA: 1s - loss: 0.0407 - accuracy: 0.9868
1004/1688 [================>.............] - ETA: 1s - loss: 0.0412 - accuracy: 0.9866
1032/1688 [=================>............] - ETA: 1s - loss: 0.0414 - accuracy: 0.9864
1060/1688 [=================>............] - ETA: 1s - loss: 0.0417 - accuracy: 0.9863
1088/1688 [==================>...........] - ETA: 1s - loss: 0.0414 - accuracy: 0.9864
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0414 - accuracy: 0.9864
1144/1688 [===================>..........] - ETA: 0s - loss: 0.0412 - accuracy: 0.9864
1172/1688 [===================>..........] - ETA: 0s - loss: 0.0416 - accuracy: 0.9863
1200/1688 [====================>.........] - ETA: 0s - loss: 0.0414 - accuracy: 0.9865
1228/1688 [====================>.........] - ETA: 0s - loss: 0.0409 - accuracy: 0.9866
1256/1688 [=====================>........] - ETA: 0s - loss: 0.0411 - accuracy: 0.9866
1284/1688 [=====================>........] - ETA: 0s - loss: 0.0414 - accuracy: 0.9865
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0413 - accuracy: 0.9865
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0416 - accuracy: 0.9865
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0418 - accuracy: 0.9864
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0421 - accuracy: 0.9864
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0419 - accuracy: 0.9864
1451/1688 [========================>.....] - ETA: 0s - loss: 0.0421 - accuracy: 0.9863
1479/1688 [=========================>....] - ETA: 0s - loss: 0.0419 - accuracy: 0.9865
1507/1688 [=========================>....] - ETA: 0s - loss: 0.0418 - accuracy: 0.9865
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0417 - accuracy: 0.9865
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0419 - accuracy: 0.9864
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0418 - accuracy: 0.9864
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0422 - accuracy: 0.9864
1646/1688 [============================>.] - ETA: 0s - loss: 0.0423 - accuracy: 0.9863
1674/1688 [============================>.] - ETA: 0s - loss: 0.0422 - accuracy: 0.9863
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0423 - accuracy: 0.9862 - val_loss: 0.0783 - val_accuracy: 0.9822
Epoch 5/10
1/1688 [..............................] - ETA: 3s - loss: 0.0289 - accuracy: 0.9688
29/1688 [..............................] - ETA: 2s - loss: 0.0364 - accuracy: 0.9892
56/1688 [..............................] - ETA: 3s - loss: 0.0283 - accuracy: 0.9905
84/1688 [>.............................] - ETA: 2s - loss: 0.0294 - accuracy: 0.9903
112/1688 [>.............................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9905
141/1688 [=>............................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9900
168/1688 [=>............................] - ETA: 2s - loss: 0.0303 - accuracy: 0.9898
196/1688 [==>...........................] - ETA: 2s - loss: 0.0293 - accuracy: 0.9896
224/1688 [==>...........................] - ETA: 2s - loss: 0.0288 - accuracy: 0.9900
252/1688 [===>..........................] - ETA: 2s - loss: 0.0283 - accuracy: 0.9905
280/1688 [===>..........................] - ETA: 2s - loss: 0.0268 - accuracy: 0.9910
308/1688 [====>.........................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9909
337/1688 [====>.........................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9905
365/1688 [=====>........................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9905
392/1688 [=====>........................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9906
420/1688 [======>.......................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9907
448/1688 [======>.......................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9903
477/1688 [=======>......................] - ETA: 2s - loss: 0.0290 - accuracy: 0.9902
504/1688 [=======>......................] - ETA: 2s - loss: 0.0293 - accuracy: 0.9901
532/1688 [========>.....................] - ETA: 2s - loss: 0.0296 - accuracy: 0.9900
559/1688 [========>.....................] - ETA: 2s - loss: 0.0300 - accuracy: 0.9900
587/1688 [=========>....................] - ETA: 2s - loss: 0.0298 - accuracy: 0.9899
614/1688 [=========>....................] - ETA: 1s - loss: 0.0297 - accuracy: 0.9899
641/1688 [==========>...................] - ETA: 1s - loss: 0.0305 - accuracy: 0.9897
669/1688 [==========>...................] - ETA: 1s - loss: 0.0307 - accuracy: 0.9896
697/1688 [===========>..................] - ETA: 1s - loss: 0.0310 - accuracy: 0.9894
725/1688 [===========>..................] - ETA: 1s - loss: 0.0311 - accuracy: 0.9895
753/1688 [============>.................] - ETA: 1s - loss: 0.0313 - accuracy: 0.9894
781/1688 [============>.................] - ETA: 1s - loss: 0.0319 - accuracy: 0.9892
808/1688 [=============>................] - ETA: 1s - loss: 0.0323 - accuracy: 0.9891
836/1688 [=============>................] - ETA: 1s - loss: 0.0323 - accuracy: 0.9890
864/1688 [==============>...............] - ETA: 1s - loss: 0.0322 - accuracy: 0.9890
892/1688 [==============>...............] - ETA: 1s - loss: 0.0329 - accuracy: 0.9888
920/1688 [===============>..............] - ETA: 1s - loss: 0.0332 - accuracy: 0.9889
948/1688 [===============>..............] - ETA: 1s - loss: 0.0334 - accuracy: 0.9889
976/1688 [================>.............] - ETA: 1s - loss: 0.0335 - accuracy: 0.9888
1004/1688 [================>.............] - ETA: 1s - loss: 0.0334 - accuracy: 0.9888
1033/1688 [=================>............] - ETA: 1s - loss: 0.0335 - accuracy: 0.9888
1061/1688 [=================>............] - ETA: 1s - loss: 0.0338 - accuracy: 0.9887
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0337 - accuracy: 0.9888
1118/1688 [==================>...........] - ETA: 1s - loss: 0.0340 - accuracy: 0.9886
1147/1688 [===================>..........] - ETA: 0s - loss: 0.0337 - accuracy: 0.9887
1174/1688 [===================>..........] - ETA: 0s - loss: 0.0339 - accuracy: 0.9887
1203/1688 [====================>.........] - ETA: 0s - loss: 0.0339 - accuracy: 0.9888
1231/1688 [====================>.........] - ETA: 0s - loss: 0.0338 - accuracy: 0.9888
1259/1688 [=====================>........] - ETA: 0s - loss: 0.0338 - accuracy: 0.9888
1287/1688 [=====================>........] - ETA: 0s - loss: 0.0339 - accuracy: 0.9889
1316/1688 [======================>.......] - ETA: 0s - loss: 0.0338 - accuracy: 0.9889
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0340 - accuracy: 0.9889
1371/1688 [=======================>......] - ETA: 0s - loss: 0.0342 - accuracy: 0.9888
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0343 - accuracy: 0.9888
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0342 - accuracy: 0.9888
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0338 - accuracy: 0.9889
1481/1688 [=========================>....] - ETA: 0s - loss: 0.0339 - accuracy: 0.9888
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0343 - accuracy: 0.9888
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0340 - accuracy: 0.9888
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0343 - accuracy: 0.9887
1592/1688 [===========================>..] - ETA: 0s - loss: 0.0343 - accuracy: 0.9887
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0342 - accuracy: 0.9887
1648/1688 [============================>.] - ETA: 0s - loss: 0.0342 - accuracy: 0.9887
1677/1688 [============================>.] - ETA: 0s - loss: 0.0340 - accuracy: 0.9887
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0338 - accuracy: 0.9888 - val_loss: 0.0573 - val_accuracy: 0.9848
Epoch 6/10
1/1688 [..............................] - ETA: 3s - loss: 0.0112 - accuracy: 1.0000
29/1688 [..............................] - ETA: 3s - loss: 0.0201 - accuracy: 0.9946
57/1688 [>.............................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9940
84/1688 [>.............................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9914
111/1688 [>.............................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9921
139/1688 [=>............................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9924
167/1688 [=>............................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9918
195/1688 [==>...........................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9918
223/1688 [==>...........................] - ETA: 2s - loss: 0.0225 - accuracy: 0.9922
250/1688 [===>..........................] - ETA: 2s - loss: 0.0239 - accuracy: 0.9918
278/1688 [===>..........................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9916
306/1688 [====>.........................] - ETA: 2s - loss: 0.0255 - accuracy: 0.9916
334/1688 [====>.........................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9916
362/1688 [=====>........................] - ETA: 2s - loss: 0.0239 - accuracy: 0.9919
390/1688 [=====>........................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9920
418/1688 [======>.......................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9922
446/1688 [======>.......................] - ETA: 2s - loss: 0.0248 - accuracy: 0.9919
474/1688 [=======>......................] - ETA: 2s - loss: 0.0252 - accuracy: 0.9914
501/1688 [=======>......................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9915
529/1688 [========>.....................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9914
557/1688 [========>.....................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9912
584/1688 [=========>....................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9913
611/1688 [=========>....................] - ETA: 1s - loss: 0.0267 - accuracy: 0.9910
638/1688 [==========>...................] - ETA: 1s - loss: 0.0268 - accuracy: 0.9911
666/1688 [==========>...................] - ETA: 1s - loss: 0.0268 - accuracy: 0.9910
694/1688 [===========>..................] - ETA: 1s - loss: 0.0266 - accuracy: 0.9911
722/1688 [===========>..................] - ETA: 1s - loss: 0.0266 - accuracy: 0.9911
750/1688 [============>.................] - ETA: 1s - loss: 0.0269 - accuracy: 0.9910
777/1688 [============>.................] - ETA: 1s - loss: 0.0267 - accuracy: 0.9910
806/1688 [=============>................] - ETA: 1s - loss: 0.0273 - accuracy: 0.9910
834/1688 [=============>................] - ETA: 1s - loss: 0.0273 - accuracy: 0.9909
862/1688 [==============>...............] - ETA: 1s - loss: 0.0273 - accuracy: 0.9908
890/1688 [==============>...............] - ETA: 1s - loss: 0.0281 - accuracy: 0.9906
918/1688 [===============>..............] - ETA: 1s - loss: 0.0281 - accuracy: 0.9906
946/1688 [===============>..............] - ETA: 1s - loss: 0.0281 - accuracy: 0.9906
974/1688 [================>.............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9905
1002/1688 [================>.............] - ETA: 1s - loss: 0.0286 - accuracy: 0.9903
1029/1688 [=================>............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9904
1057/1688 [=================>............] - ETA: 1s - loss: 0.0282 - accuracy: 0.9905
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0280 - accuracy: 0.9906
1113/1688 [==================>...........] - ETA: 1s - loss: 0.0277 - accuracy: 0.9906
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0275 - accuracy: 0.9907
1168/1688 [===================>..........] - ETA: 0s - loss: 0.0274 - accuracy: 0.9907
1196/1688 [====================>.........] - ETA: 0s - loss: 0.0284 - accuracy: 0.9904
1224/1688 [====================>.........] - ETA: 0s - loss: 0.0285 - accuracy: 0.9903
1252/1688 [=====================>........] - ETA: 0s - loss: 0.0283 - accuracy: 0.9905
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0285 - accuracy: 0.9904
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0286 - accuracy: 0.9903
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0284 - accuracy: 0.9903
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9901
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0287 - accuracy: 0.9901
1418/1688 [========================>.....] - ETA: 0s - loss: 0.0286 - accuracy: 0.9901
1446/1688 [========================>.....] - ETA: 0s - loss: 0.0288 - accuracy: 0.9901
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0285 - accuracy: 0.9902
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0285 - accuracy: 0.9902
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0286 - accuracy: 0.9901
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0289 - accuracy: 0.9901
1585/1688 [===========================>..] - ETA: 0s - loss: 0.0290 - accuracy: 0.9900
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0293 - accuracy: 0.9900
1641/1688 [============================>.] - ETA: 0s - loss: 0.0291 - accuracy: 0.9900
1669/1688 [============================>.] - ETA: 0s - loss: 0.0294 - accuracy: 0.9900
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0294 - accuracy: 0.9899 - val_loss: 0.0702 - val_accuracy: 0.9838
Epoch 7/10
1/1688 [..............................] - ETA: 3s - loss: 0.0154 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9948
57/1688 [>.............................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9923
84/1688 [>.............................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9914
111/1688 [>.............................] - ETA: 2s - loss: 0.0241 - accuracy: 0.9916
139/1688 [=>............................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9919
167/1688 [=>............................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9918
195/1688 [==>...........................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9925
223/1688 [==>...........................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9926
250/1688 [===>..........................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9927
278/1688 [===>..........................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9929
306/1688 [====>.........................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9929
334/1688 [====>.........................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9930
362/1688 [=====>........................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9929
390/1688 [=====>........................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9931
418/1688 [======>.......................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9933
446/1688 [======>.......................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9933
474/1688 [=======>......................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9937
502/1688 [=======>......................] - ETA: 2s - loss: 0.0194 - accuracy: 0.9935
530/1688 [========>.....................] - ETA: 2s - loss: 0.0197 - accuracy: 0.9935
558/1688 [========>.....................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9936
586/1688 [=========>....................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9936
614/1688 [=========>....................] - ETA: 1s - loss: 0.0198 - accuracy: 0.9936
642/1688 [==========>...................] - ETA: 1s - loss: 0.0195 - accuracy: 0.9938
670/1688 [==========>...................] - ETA: 1s - loss: 0.0194 - accuracy: 0.9940
698/1688 [===========>..................] - ETA: 1s - loss: 0.0204 - accuracy: 0.9938
726/1688 [===========>..................] - ETA: 1s - loss: 0.0207 - accuracy: 0.9937
754/1688 [============>.................] - ETA: 1s - loss: 0.0209 - accuracy: 0.9936
782/1688 [============>.................] - ETA: 1s - loss: 0.0206 - accuracy: 0.9936
810/1688 [=============>................] - ETA: 1s - loss: 0.0205 - accuracy: 0.9936
838/1688 [=============>................] - ETA: 1s - loss: 0.0208 - accuracy: 0.9934
866/1688 [==============>...............] - ETA: 1s - loss: 0.0207 - accuracy: 0.9934
893/1688 [==============>...............] - ETA: 1s - loss: 0.0213 - accuracy: 0.9933
920/1688 [===============>..............] - ETA: 1s - loss: 0.0218 - accuracy: 0.9931
948/1688 [===============>..............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9932
976/1688 [================>.............] - ETA: 1s - loss: 0.0222 - accuracy: 0.9930
1004/1688 [================>.............] - ETA: 1s - loss: 0.0224 - accuracy: 0.9929
1033/1688 [=================>............] - ETA: 1s - loss: 0.0224 - accuracy: 0.9930
1060/1688 [=================>............] - ETA: 1s - loss: 0.0229 - accuracy: 0.9929
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0229 - accuracy: 0.9929
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0226 - accuracy: 0.9930
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0224 - accuracy: 0.9930
1173/1688 [===================>..........] - ETA: 0s - loss: 0.0224 - accuracy: 0.9930
1201/1688 [====================>.........] - ETA: 0s - loss: 0.0225 - accuracy: 0.9929
1229/1688 [====================>.........] - ETA: 0s - loss: 0.0222 - accuracy: 0.9931
1257/1688 [=====================>........] - ETA: 0s - loss: 0.0224 - accuracy: 0.9931
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0226 - accuracy: 0.9929
1313/1688 [======================>.......] - ETA: 0s - loss: 0.0226 - accuracy: 0.9929
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0227 - accuracy: 0.9929
1368/1688 [=======================>......] - ETA: 0s - loss: 0.0227 - accuracy: 0.9929
1396/1688 [=======================>......] - ETA: 0s - loss: 0.0230 - accuracy: 0.9928
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0232 - accuracy: 0.9927
1451/1688 [========================>.....] - ETA: 0s - loss: 0.0234 - accuracy: 0.9925
1479/1688 [=========================>....] - ETA: 0s - loss: 0.0235 - accuracy: 0.9925
1506/1688 [=========================>....] - ETA: 0s - loss: 0.0236 - accuracy: 0.9924
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0235 - accuracy: 0.9925
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0236 - accuracy: 0.9925
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0237 - accuracy: 0.9924
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0236 - accuracy: 0.9925
1645/1688 [============================>.] - ETA: 0s - loss: 0.0236 - accuracy: 0.9925
1673/1688 [============================>.] - ETA: 0s - loss: 0.0237 - accuracy: 0.9924
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0238 - accuracy: 0.9924 - val_loss: 0.0751 - val_accuracy: 0.9810
Epoch 8/10
1/1688 [..............................] - ETA: 3s - loss: 0.0043 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9937
58/1688 [>.............................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9957
86/1688 [>.............................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9964
114/1688 [=>............................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9953
143/1688 [=>............................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9948
171/1688 [==>...........................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9943
199/1688 [==>...........................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9945
227/1688 [===>..........................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9942
255/1688 [===>..........................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9941
283/1688 [====>.........................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9941
311/1688 [====>.........................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9942
339/1688 [=====>........................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9944
366/1688 [=====>........................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9947
394/1688 [======>.......................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9949
422/1688 [======>.......................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9950
449/1688 [======>.......................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9951
476/1688 [=======>......................] - ETA: 2s - loss: 0.0137 - accuracy: 0.9952
504/1688 [=======>......................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9948
532/1688 [========>.....................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9947
560/1688 [========>.....................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9945
587/1688 [=========>....................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9945
615/1688 [=========>....................] - ETA: 1s - loss: 0.0147 - accuracy: 0.9945
643/1688 [==========>...................] - ETA: 1s - loss: 0.0146 - accuracy: 0.9946
671/1688 [==========>...................] - ETA: 1s - loss: 0.0150 - accuracy: 0.9944
699/1688 [===========>..................] - ETA: 1s - loss: 0.0156 - accuracy: 0.9941
727/1688 [===========>..................] - ETA: 1s - loss: 0.0162 - accuracy: 0.9940
755/1688 [============>.................] - ETA: 1s - loss: 0.0165 - accuracy: 0.9940
783/1688 [============>.................] - ETA: 1s - loss: 0.0168 - accuracy: 0.9939
811/1688 [=============>................] - ETA: 1s - loss: 0.0166 - accuracy: 0.9939
839/1688 [=============>................] - ETA: 1s - loss: 0.0166 - accuracy: 0.9940
868/1688 [==============>...............] - ETA: 1s - loss: 0.0164 - accuracy: 0.9941
896/1688 [==============>...............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9941
924/1688 [===============>..............] - ETA: 1s - loss: 0.0167 - accuracy: 0.9940
952/1688 [===============>..............] - ETA: 1s - loss: 0.0169 - accuracy: 0.9938
980/1688 [================>.............] - ETA: 1s - loss: 0.0171 - accuracy: 0.9938
1008/1688 [================>.............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9936
1036/1688 [=================>............] - ETA: 1s - loss: 0.0180 - accuracy: 0.9934
1064/1688 [=================>............] - ETA: 1s - loss: 0.0183 - accuracy: 0.9933
1092/1688 [==================>...........] - ETA: 1s - loss: 0.0184 - accuracy: 0.9932
1119/1688 [==================>...........] - ETA: 1s - loss: 0.0189 - accuracy: 0.9931
1147/1688 [===================>..........] - ETA: 0s - loss: 0.0191 - accuracy: 0.9930
1174/1688 [===================>..........] - ETA: 0s - loss: 0.0192 - accuracy: 0.9929
1202/1688 [====================>.........] - ETA: 0s - loss: 0.0194 - accuracy: 0.9929
1230/1688 [====================>.........] - ETA: 0s - loss: 0.0197 - accuracy: 0.9927
1258/1688 [=====================>........] - ETA: 0s - loss: 0.0199 - accuracy: 0.9926
1286/1688 [=====================>........] - ETA: 0s - loss: 0.0203 - accuracy: 0.9925
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0206 - accuracy: 0.9925
1342/1688 [======================>.......] - ETA: 0s - loss: 0.0208 - accuracy: 0.9924
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0209 - accuracy: 0.9924
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9923
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0216 - accuracy: 0.9922
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0214 - accuracy: 0.9923
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0215 - accuracy: 0.9923
1510/1688 [=========================>....] - ETA: 0s - loss: 0.0214 - accuracy: 0.9923
1538/1688 [==========================>...] - ETA: 0s - loss: 0.0215 - accuracy: 0.9923
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0214 - accuracy: 0.9924
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9923
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0215 - accuracy: 0.9924
1650/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9923
1678/1688 [============================>.] - ETA: 0s - loss: 0.0219 - accuracy: 0.9922
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0218 - accuracy: 0.9922 - val_loss: 0.0525 - val_accuracy: 0.9877
Epoch 9/10
1/1688 [..............................] - ETA: 3s - loss: 0.0014 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9969
58/1688 [>.............................] - ETA: 2s - loss: 0.0082 - accuracy: 0.9973
86/1688 [>.............................] - ETA: 2s - loss: 0.0077 - accuracy: 0.9978
114/1688 [=>............................] - ETA: 2s - loss: 0.0081 - accuracy: 0.9975
142/1688 [=>............................] - ETA: 2s - loss: 0.0089 - accuracy: 0.9974
169/1688 [==>...........................] - ETA: 2s - loss: 0.0092 - accuracy: 0.9974
196/1688 [==>...........................] - ETA: 2s - loss: 0.0088 - accuracy: 0.9973
224/1688 [==>...........................] - ETA: 2s - loss: 0.0083 - accuracy: 0.9975
252/1688 [===>..........................] - ETA: 2s - loss: 0.0079 - accuracy: 0.9978
280/1688 [===>..........................] - ETA: 2s - loss: 0.0076 - accuracy: 0.9980
309/1688 [====>.........................] - ETA: 2s - loss: 0.0076 - accuracy: 0.9981
337/1688 [====>.........................] - ETA: 2s - loss: 0.0078 - accuracy: 0.9981
364/1688 [=====>........................] - ETA: 2s - loss: 0.0083 - accuracy: 0.9979
392/1688 [=====>........................] - ETA: 2s - loss: 0.0090 - accuracy: 0.9974
420/1688 [======>.......................] - ETA: 2s - loss: 0.0092 - accuracy: 0.9973
448/1688 [======>.......................] - ETA: 2s - loss: 0.0091 - accuracy: 0.9973
476/1688 [=======>......................] - ETA: 2s - loss: 0.0092 - accuracy: 0.9972
504/1688 [=======>......................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9971
531/1688 [========>.....................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9971
559/1688 [========>.....................] - ETA: 2s - loss: 0.0098 - accuracy: 0.9969
587/1688 [=========>....................] - ETA: 2s - loss: 0.0101 - accuracy: 0.9968
614/1688 [=========>....................] - ETA: 1s - loss: 0.0101 - accuracy: 0.9967
642/1688 [==========>...................] - ETA: 1s - loss: 0.0102 - accuracy: 0.9966
671/1688 [==========>...................] - ETA: 1s - loss: 0.0106 - accuracy: 0.9965
699/1688 [===========>..................] - ETA: 1s - loss: 0.0106 - accuracy: 0.9964
726/1688 [===========>..................] - ETA: 1s - loss: 0.0109 - accuracy: 0.9963
754/1688 [============>.................] - ETA: 1s - loss: 0.0114 - accuracy: 0.9963
782/1688 [============>.................] - ETA: 1s - loss: 0.0115 - accuracy: 0.9962
810/1688 [=============>................] - ETA: 1s - loss: 0.0118 - accuracy: 0.9960
839/1688 [=============>................] - ETA: 1s - loss: 0.0121 - accuracy: 0.9959
867/1688 [==============>...............] - ETA: 1s - loss: 0.0127 - accuracy: 0.9958
895/1688 [==============>...............] - ETA: 1s - loss: 0.0128 - accuracy: 0.9957
923/1688 [===============>..............] - ETA: 1s - loss: 0.0134 - accuracy: 0.9955
951/1688 [===============>..............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9954
980/1688 [================>.............] - ETA: 1s - loss: 0.0137 - accuracy: 0.9954
1008/1688 [================>.............] - ETA: 1s - loss: 0.0138 - accuracy: 0.9953
1036/1688 [=================>............] - ETA: 1s - loss: 0.0143 - accuracy: 0.9953
1062/1688 [=================>............] - ETA: 1s - loss: 0.0145 - accuracy: 0.9952
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0146 - accuracy: 0.9953
1118/1688 [==================>...........] - ETA: 1s - loss: 0.0147 - accuracy: 0.9953
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0147 - accuracy: 0.9953
1173/1688 [===================>..........] - ETA: 0s - loss: 0.0149 - accuracy: 0.9951
1200/1688 [====================>.........] - ETA: 0s - loss: 0.0152 - accuracy: 0.9951
1227/1688 [====================>.........] - ETA: 0s - loss: 0.0151 - accuracy: 0.9951
1256/1688 [=====================>........] - ETA: 0s - loss: 0.0152 - accuracy: 0.9950
1284/1688 [=====================>........] - ETA: 0s - loss: 0.0153 - accuracy: 0.9950
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0152 - accuracy: 0.9950
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0155 - accuracy: 0.9949
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0155 - accuracy: 0.9949
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0157 - accuracy: 0.9948
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9948
1450/1688 [========================>.....] - ETA: 0s - loss: 0.0159 - accuracy: 0.9947
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1506/1688 [=========================>....] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0161 - accuracy: 0.9948
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0163 - accuracy: 0.9947
1617/1688 [===========================>..] - ETA: 0s - loss: 0.0163 - accuracy: 0.9947
1645/1688 [============================>.] - ETA: 0s - loss: 0.0161 - accuracy: 0.9948
1673/1688 [============================>.] - ETA: 0s - loss: 0.0162 - accuracy: 0.9948
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0163 - accuracy: 0.9947 - val_loss: 0.0787 - val_accuracy: 0.9823
Epoch 10/10
1/1688 [..............................] - ETA: 3s - loss: 0.0023 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0158 - accuracy: 0.9958
57/1688 [>.............................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9956
85/1688 [>.............................] - ETA: 2s - loss: 0.0120 - accuracy: 0.9960
113/1688 [=>............................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9961
141/1688 [=>............................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9965
169/1688 [==>...........................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9965
197/1688 [==>...........................] - ETA: 2s - loss: 0.0100 - accuracy: 0.9970
225/1688 [==>...........................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9971
253/1688 [===>..........................] - ETA: 2s - loss: 0.0103 - accuracy: 0.9968
281/1688 [===>..........................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9966
309/1688 [====>.........................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9967
337/1688 [====>.........................] - ETA: 2s - loss: 0.0114 - accuracy: 0.9966
364/1688 [=====>........................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9967
392/1688 [=====>........................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9966
420/1688 [======>.......................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9966
448/1688 [======>.......................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9966
476/1688 [=======>......................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9966
504/1688 [=======>......................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9966
532/1688 [========>.....................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9965
560/1688 [========>.....................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9966
588/1688 [=========>....................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9964
616/1688 [=========>....................] - ETA: 1s - loss: 0.0111 - accuracy: 0.9964
644/1688 [==========>...................] - ETA: 1s - loss: 0.0114 - accuracy: 0.9962
672/1688 [==========>...................] - ETA: 1s - loss: 0.0113 - accuracy: 0.9961
699/1688 [===========>..................] - ETA: 1s - loss: 0.0116 - accuracy: 0.9960
727/1688 [===========>..................] - ETA: 1s - loss: 0.0119 - accuracy: 0.9958
754/1688 [============>.................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9956
782/1688 [============>.................] - ETA: 1s - loss: 0.0126 - accuracy: 0.9956
810/1688 [=============>................] - ETA: 1s - loss: 0.0126 - accuracy: 0.9956
838/1688 [=============>................] - ETA: 1s - loss: 0.0126 - accuracy: 0.9956
866/1688 [==============>...............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9957
894/1688 [==============>...............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9956
922/1688 [===============>..............] - ETA: 1s - loss: 0.0125 - accuracy: 0.9956
950/1688 [===============>..............] - ETA: 1s - loss: 0.0127 - accuracy: 0.9956
978/1688 [================>.............] - ETA: 1s - loss: 0.0130 - accuracy: 0.9956
1006/1688 [================>.............] - ETA: 1s - loss: 0.0131 - accuracy: 0.9955
1033/1688 [=================>............] - ETA: 1s - loss: 0.0131 - accuracy: 0.9955
1061/1688 [=================>............] - ETA: 1s - loss: 0.0130 - accuracy: 0.9956
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0136 - accuracy: 0.9954
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0138 - accuracy: 0.9953
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0139 - accuracy: 0.9952
1173/1688 [===================>..........] - ETA: 0s - loss: 0.0139 - accuracy: 0.9952
1201/1688 [====================>.........] - ETA: 0s - loss: 0.0137 - accuracy: 0.9953
1229/1688 [====================>.........] - ETA: 0s - loss: 0.0140 - accuracy: 0.9952
1257/1688 [=====================>........] - ETA: 0s - loss: 0.0142 - accuracy: 0.9950
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0144 - accuracy: 0.9950
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0145 - accuracy: 0.9949
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0144 - accuracy: 0.9949
1368/1688 [=======================>......] - ETA: 0s - loss: 0.0143 - accuracy: 0.9950
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0143 - accuracy: 0.9950
1422/1688 [========================>.....] - ETA: 0s - loss: 0.0143 - accuracy: 0.9949
1450/1688 [========================>.....] - ETA: 0s - loss: 0.0144 - accuracy: 0.9949
1478/1688 [=========================>....] - ETA: 0s - loss: 0.0148 - accuracy: 0.9948
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0149 - accuracy: 0.9948
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0148 - accuracy: 0.9948
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0148 - accuracy: 0.9948
1589/1688 [===========================>..] - ETA: 0s - loss: 0.0147 - accuracy: 0.9948
1616/1688 [===========================>..] - ETA: 0s - loss: 0.0148 - accuracy: 0.9948
1644/1688 [============================>.] - ETA: 0s - loss: 0.0147 - accuracy: 0.9949
1672/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9948
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0147 - accuracy: 0.9949 - val_loss: 0.0645 - val_accuracy: 0.9865
Test accuracy: 0.983299970626831
4. Model quantization
We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.
For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).
The quantized model is a Keras model where the neural layers (Conv2D, Dense)
and the ReLU layers are replaced with custom CNN2SNN quantized layers
(QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions
can be applied on this new model: summary(), compile(), fit(). etc.
Note
The quantize function folds the batch normalization layers into
the corresponding neural layer. The new weights are computed
according to this folding operation.
from cnn2snn import quantize
model_quantized = quantize(model_keras,
input_weight_quantization=8,
weight_quantization=4,
activ_quantization=4)
model_quantized.summary()
Model: "sequential_49"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (QuantizedConv2D) (None, 13, 13, 32) 320
re_lu (QuantizedReLU) (None, 13, 13, 32) 0
separable_conv2d (Quantized (None, 7, 7, 64) 2400
SeparableConv2D)
re_lu_1 (QuantizedReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (QuantizedDense) (None, 10) 31370
=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________
Check the quantized model accuracy.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])
Test accuracy after 8-4-4 quantization: 0.8206999897956848
Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.
Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.
Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.
from cnn2snn import quantize_layer
model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])
# To recover the original model accuracy, a quantization-aware training phase
# is required.
Test accuracy after low bitwidth quantization: 0.11349999904632568
5. Model fine tuning (quantization-aware training)
This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.
Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])
Epoch 1/5
1/1688 [..............................] - ETA: 19:39 - loss: 2.3000 - accuracy: 0.1875
20/1688 [..............................] - ETA: 4s - loss: 2.3017 - accuracy: 0.1187
39/1688 [..............................] - ETA: 4s - loss: 2.3017 - accuracy: 0.1082
59/1688 [>.............................] - ETA: 4s - loss: 2.3080 - accuracy: 0.1102
78/1688 [>.............................] - ETA: 4s - loss: 2.3262 - accuracy: 0.1078
97/1688 [>.............................] - ETA: 4s - loss: 2.3631 - accuracy: 0.1221
116/1688 [=>............................] - ETA: 4s - loss: 2.3798 - accuracy: 0.1460
135/1688 [=>............................] - ETA: 4s - loss: 2.4306 - accuracy: 0.1806
155/1688 [=>............................] - ETA: 4s - loss: 2.4210 - accuracy: 0.2333
174/1688 [==>...........................] - ETA: 4s - loss: 2.3677 - accuracy: 0.2857
194/1688 [==>...........................] - ETA: 3s - loss: 2.3051 - accuracy: 0.3352
213/1688 [==>...........................] - ETA: 3s - loss: 2.2576 - accuracy: 0.3773
233/1688 [===>..........................] - ETA: 3s - loss: 2.1722 - accuracy: 0.4193
252/1688 [===>..........................] - ETA: 3s - loss: 2.0905 - accuracy: 0.4531
272/1688 [===>..........................] - ETA: 3s - loss: 2.0094 - accuracy: 0.4846
291/1688 [====>.........................] - ETA: 3s - loss: 1.9609 - accuracy: 0.5091
311/1688 [====>.........................] - ETA: 3s - loss: 1.9118 - accuracy: 0.5331
330/1688 [====>.........................] - ETA: 3s - loss: 1.8672 - accuracy: 0.5544
350/1688 [=====>........................] - ETA: 3s - loss: 1.8251 - accuracy: 0.5738
369/1688 [=====>........................] - ETA: 3s - loss: 1.8218 - accuracy: 0.5899
389/1688 [=====>........................] - ETA: 3s - loss: 1.7892 - accuracy: 0.6063
408/1688 [======>.......................] - ETA: 3s - loss: 1.7602 - accuracy: 0.6209
428/1688 [======>.......................] - ETA: 3s - loss: 1.7295 - accuracy: 0.6349
447/1688 [======>.......................] - ETA: 3s - loss: 1.7054 - accuracy: 0.6470
466/1688 [=======>......................] - ETA: 3s - loss: 1.6685 - accuracy: 0.6591
485/1688 [=======>......................] - ETA: 3s - loss: 1.6381 - accuracy: 0.6700
504/1688 [=======>......................] - ETA: 3s - loss: 1.6068 - accuracy: 0.6806
523/1688 [========>.....................] - ETA: 3s - loss: 1.5761 - accuracy: 0.6901
542/1688 [========>.....................] - ETA: 3s - loss: 1.5610 - accuracy: 0.6983
561/1688 [========>.....................] - ETA: 2s - loss: 1.5343 - accuracy: 0.7067
581/1688 [=========>....................] - ETA: 2s - loss: 1.5007 - accuracy: 0.7147
600/1688 [=========>....................] - ETA: 2s - loss: 1.4759 - accuracy: 0.7220
619/1688 [==========>...................] - ETA: 2s - loss: 1.4665 - accuracy: 0.7280
639/1688 [==========>...................] - ETA: 2s - loss: 1.4440 - accuracy: 0.7348
658/1688 [==========>...................] - ETA: 2s - loss: 1.4332 - accuracy: 0.7405
677/1688 [===========>..................] - ETA: 2s - loss: 1.4211 - accuracy: 0.7459
697/1688 [===========>..................] - ETA: 2s - loss: 1.4064 - accuracy: 0.7515
716/1688 [===========>..................] - ETA: 2s - loss: 1.3907 - accuracy: 0.7562
736/1688 [============>.................] - ETA: 2s - loss: 1.3783 - accuracy: 0.7612
755/1688 [============>.................] - ETA: 2s - loss: 1.3643 - accuracy: 0.7656
775/1688 [============>.................] - ETA: 2s - loss: 1.3442 - accuracy: 0.7704
794/1688 [=============>................] - ETA: 2s - loss: 1.3336 - accuracy: 0.7744
813/1688 [=============>................] - ETA: 2s - loss: 1.3258 - accuracy: 0.7782
832/1688 [=============>................] - ETA: 2s - loss: 1.3050 - accuracy: 0.7825
852/1688 [==============>...............] - ETA: 2s - loss: 1.2913 - accuracy: 0.7864
872/1688 [==============>...............] - ETA: 2s - loss: 1.2802 - accuracy: 0.7898
891/1688 [==============>...............] - ETA: 2s - loss: 1.2637 - accuracy: 0.7935
910/1688 [===============>..............] - ETA: 2s - loss: 1.2497 - accuracy: 0.7968
929/1688 [===============>..............] - ETA: 2s - loss: 1.2398 - accuracy: 0.7998
948/1688 [===============>..............] - ETA: 1s - loss: 1.2278 - accuracy: 0.8030
967/1688 [================>.............] - ETA: 1s - loss: 1.2218 - accuracy: 0.8058
986/1688 [================>.............] - ETA: 1s - loss: 1.2131 - accuracy: 0.8086
1006/1688 [================>.............] - ETA: 1s - loss: 1.1982 - accuracy: 0.8115
1026/1688 [=================>............] - ETA: 1s - loss: 1.1890 - accuracy: 0.8143
1045/1688 [=================>............] - ETA: 1s - loss: 1.1802 - accuracy: 0.8168
1064/1688 [=================>............] - ETA: 1s - loss: 1.1723 - accuracy: 0.8194
1083/1688 [==================>...........] - ETA: 1s - loss: 1.1662 - accuracy: 0.8215
1103/1688 [==================>...........] - ETA: 1s - loss: 1.1537 - accuracy: 0.8241
1122/1688 [==================>...........] - ETA: 1s - loss: 1.1482 - accuracy: 0.8264
1141/1688 [===================>..........] - ETA: 1s - loss: 1.1438 - accuracy: 0.8284
1161/1688 [===================>..........] - ETA: 1s - loss: 1.1391 - accuracy: 0.8306
1180/1688 [===================>..........] - ETA: 1s - loss: 1.1265 - accuracy: 0.8328
1200/1688 [====================>.........] - ETA: 1s - loss: 1.1205 - accuracy: 0.8347
1220/1688 [====================>.........] - ETA: 1s - loss: 1.1150 - accuracy: 0.8367
1239/1688 [=====================>........] - ETA: 1s - loss: 1.1084 - accuracy: 0.8387
1258/1688 [=====================>........] - ETA: 1s - loss: 1.1002 - accuracy: 0.8405
1278/1688 [=====================>........] - ETA: 1s - loss: 1.1017 - accuracy: 0.8421
1297/1688 [======================>.......] - ETA: 1s - loss: 1.0967 - accuracy: 0.8438
1316/1688 [======================>.......] - ETA: 0s - loss: 1.0901 - accuracy: 0.8454
1335/1688 [======================>.......] - ETA: 0s - loss: 1.0866 - accuracy: 0.8469
1354/1688 [=======================>......] - ETA: 0s - loss: 1.0849 - accuracy: 0.8483
1373/1688 [=======================>......] - ETA: 0s - loss: 1.0824 - accuracy: 0.8497
1393/1688 [=======================>......] - ETA: 0s - loss: 1.0740 - accuracy: 0.8514
1412/1688 [========================>.....] - ETA: 0s - loss: 1.0633 - accuracy: 0.8531
1432/1688 [========================>.....] - ETA: 0s - loss: 1.0620 - accuracy: 0.8545
1452/1688 [========================>.....] - ETA: 0s - loss: 1.0578 - accuracy: 0.8559
1471/1688 [=========================>....] - ETA: 0s - loss: 1.0522 - accuracy: 0.8570
1490/1688 [=========================>....] - ETA: 0s - loss: 1.0457 - accuracy: 0.8583
1509/1688 [=========================>....] - ETA: 0s - loss: 1.0380 - accuracy: 0.8596
1528/1688 [==========================>...] - ETA: 0s - loss: 1.0346 - accuracy: 0.8608
1548/1688 [==========================>...] - ETA: 0s - loss: 1.0270 - accuracy: 0.8622
1567/1688 [==========================>...] - ETA: 0s - loss: 1.0225 - accuracy: 0.8634
1587/1688 [===========================>..] - ETA: 0s - loss: 1.0174 - accuracy: 0.8646
1607/1688 [===========================>..] - ETA: 0s - loss: 1.0115 - accuracy: 0.8659
1627/1688 [===========================>..] - ETA: 0s - loss: 1.0096 - accuracy: 0.8669
1647/1688 [============================>.] - ETA: 0s - loss: 1.0051 - accuracy: 0.8681
1666/1688 [============================>.] - ETA: 0s - loss: 1.0022 - accuracy: 0.8690
1685/1688 [============================>.] - ETA: 0s - loss: 0.9990 - accuracy: 0.8700
1688/1688 [==============================] - 5s 3ms/step - loss: 0.9981 - accuracy: 0.8702 - val_loss: 0.5859 - val_accuracy: 0.9673
Epoch 2/5
1/1688 [..............................] - ETA: 4s - loss: 0.6133 - accuracy: 0.9375
21/1688 [..............................] - ETA: 4s - loss: 0.8407 - accuracy: 0.9539
40/1688 [..............................] - ETA: 4s - loss: 0.6723 - accuracy: 0.9594
60/1688 [>.............................] - ETA: 4s - loss: 0.5895 - accuracy: 0.9641
79/1688 [>.............................] - ETA: 4s - loss: 0.6286 - accuracy: 0.9616
98/1688 [>.............................] - ETA: 4s - loss: 0.6090 - accuracy: 0.9614
117/1688 [=>............................] - ETA: 4s - loss: 0.5854 - accuracy: 0.9607
137/1688 [=>............................] - ETA: 4s - loss: 0.5532 - accuracy: 0.9630
157/1688 [=>............................] - ETA: 4s - loss: 0.5791 - accuracy: 0.9624
177/1688 [==>...........................] - ETA: 3s - loss: 0.5777 - accuracy: 0.9615
196/1688 [==>...........................] - ETA: 3s - loss: 0.5820 - accuracy: 0.9621
215/1688 [==>...........................] - ETA: 3s - loss: 0.6015 - accuracy: 0.9609
235/1688 [===>..........................] - ETA: 3s - loss: 0.6362 - accuracy: 0.9601
254/1688 [===>..........................] - ETA: 3s - loss: 0.6494 - accuracy: 0.9601
273/1688 [===>..........................] - ETA: 3s - loss: 0.6472 - accuracy: 0.9602
293/1688 [====>.........................] - ETA: 3s - loss: 0.6667 - accuracy: 0.9596
312/1688 [====>.........................] - ETA: 3s - loss: 0.6696 - accuracy: 0.9600
332/1688 [====>.........................] - ETA: 3s - loss: 0.6820 - accuracy: 0.9600
352/1688 [=====>........................] - ETA: 3s - loss: 0.6736 - accuracy: 0.9602
371/1688 [=====>........................] - ETA: 3s - loss: 0.6820 - accuracy: 0.9599
391/1688 [=====>........................] - ETA: 3s - loss: 0.6734 - accuracy: 0.9603
411/1688 [======>.......................] - ETA: 3s - loss: 0.6581 - accuracy: 0.9612
430/1688 [======>.......................] - ETA: 3s - loss: 0.6535 - accuracy: 0.9614
450/1688 [======>.......................] - ETA: 3s - loss: 0.6367 - accuracy: 0.9622
469/1688 [=======>......................] - ETA: 3s - loss: 0.6341 - accuracy: 0.9625
488/1688 [=======>......................] - ETA: 3s - loss: 0.6364 - accuracy: 0.9622
508/1688 [========>.....................] - ETA: 3s - loss: 0.6298 - accuracy: 0.9624
527/1688 [========>.....................] - ETA: 3s - loss: 0.6222 - accuracy: 0.9628
547/1688 [========>.....................] - ETA: 3s - loss: 0.6191 - accuracy: 0.9626
566/1688 [=========>....................] - ETA: 2s - loss: 0.6181 - accuracy: 0.9630
586/1688 [=========>....................] - ETA: 2s - loss: 0.6188 - accuracy: 0.9628
605/1688 [=========>....................] - ETA: 2s - loss: 0.6156 - accuracy: 0.9631
624/1688 [==========>...................] - ETA: 2s - loss: 0.6127 - accuracy: 0.9634
643/1688 [==========>...................] - ETA: 2s - loss: 0.6210 - accuracy: 0.9632
661/1688 [==========>...................] - ETA: 2s - loss: 0.6228 - accuracy: 0.9634
680/1688 [===========>..................] - ETA: 2s - loss: 0.6282 - accuracy: 0.9630
700/1688 [===========>..................] - ETA: 2s - loss: 0.6267 - accuracy: 0.9632
719/1688 [===========>..................] - ETA: 2s - loss: 0.6291 - accuracy: 0.9631
739/1688 [============>.................] - ETA: 2s - loss: 0.6345 - accuracy: 0.9630
759/1688 [============>.................] - ETA: 2s - loss: 0.6520 - accuracy: 0.9627
779/1688 [============>.................] - ETA: 2s - loss: 0.6519 - accuracy: 0.9628
799/1688 [=============>................] - ETA: 2s - loss: 0.6501 - accuracy: 0.9629
818/1688 [=============>................] - ETA: 2s - loss: 0.6500 - accuracy: 0.9628
838/1688 [=============>................] - ETA: 2s - loss: 0.6425 - accuracy: 0.9631
858/1688 [==============>...............] - ETA: 2s - loss: 0.6354 - accuracy: 0.9636
877/1688 [==============>...............] - ETA: 2s - loss: 0.6411 - accuracy: 0.9636
896/1688 [==============>...............] - ETA: 2s - loss: 0.6359 - accuracy: 0.9636
916/1688 [===============>..............] - ETA: 2s - loss: 0.6381 - accuracy: 0.9634
935/1688 [===============>..............] - ETA: 1s - loss: 0.6364 - accuracy: 0.9635
954/1688 [===============>..............] - ETA: 1s - loss: 0.6325 - accuracy: 0.9637
973/1688 [================>.............] - ETA: 1s - loss: 0.6254 - accuracy: 0.9641
992/1688 [================>.............] - ETA: 1s - loss: 0.6268 - accuracy: 0.9640
1011/1688 [================>.............] - ETA: 1s - loss: 0.6349 - accuracy: 0.9637
1030/1688 [=================>............] - ETA: 1s - loss: 0.6352 - accuracy: 0.9637
1049/1688 [=================>............] - ETA: 1s - loss: 0.6329 - accuracy: 0.9639
1069/1688 [=================>............] - ETA: 1s - loss: 0.6285 - accuracy: 0.9641
1088/1688 [==================>...........] - ETA: 1s - loss: 0.6248 - accuracy: 0.9644
1107/1688 [==================>...........] - ETA: 1s - loss: 0.6237 - accuracy: 0.9645
1126/1688 [===================>..........] - ETA: 1s - loss: 0.6162 - accuracy: 0.9648
1146/1688 [===================>..........] - ETA: 1s - loss: 0.6120 - accuracy: 0.9652
1165/1688 [===================>..........] - ETA: 1s - loss: 0.6224 - accuracy: 0.9651
1184/1688 [====================>.........] - ETA: 1s - loss: 0.6220 - accuracy: 0.9651
1204/1688 [====================>.........] - ETA: 1s - loss: 0.6249 - accuracy: 0.9651
1223/1688 [====================>.........] - ETA: 1s - loss: 0.6239 - accuracy: 0.9652
1242/1688 [=====================>........] - ETA: 1s - loss: 0.6195 - accuracy: 0.9654
1261/1688 [=====================>........] - ETA: 1s - loss: 0.6193 - accuracy: 0.9655
1280/1688 [=====================>........] - ETA: 1s - loss: 0.6195 - accuracy: 0.9654
1299/1688 [======================>.......] - ETA: 1s - loss: 0.6193 - accuracy: 0.9652
1318/1688 [======================>.......] - ETA: 0s - loss: 0.6211 - accuracy: 0.9652
1337/1688 [======================>.......] - ETA: 0s - loss: 0.6249 - accuracy: 0.9651
1356/1688 [=======================>......] - ETA: 0s - loss: 0.6245 - accuracy: 0.9651
1376/1688 [=======================>......] - ETA: 0s - loss: 0.6243 - accuracy: 0.9652
1396/1688 [=======================>......] - ETA: 0s - loss: 0.6253 - accuracy: 0.9652
1416/1688 [========================>.....] - ETA: 0s - loss: 0.6254 - accuracy: 0.9653
1436/1688 [========================>.....] - ETA: 0s - loss: 0.6234 - accuracy: 0.9654
1455/1688 [========================>.....] - ETA: 0s - loss: 0.6211 - accuracy: 0.9655
1474/1688 [=========================>....] - ETA: 0s - loss: 0.6187 - accuracy: 0.9656
1493/1688 [=========================>....] - ETA: 0s - loss: 0.6166 - accuracy: 0.9658
1513/1688 [=========================>....] - ETA: 0s - loss: 0.6144 - accuracy: 0.9659
1533/1688 [==========================>...] - ETA: 0s - loss: 0.6193 - accuracy: 0.9657
1552/1688 [==========================>...] - ETA: 0s - loss: 0.6190 - accuracy: 0.9658
1571/1688 [==========================>...] - ETA: 0s - loss: 0.6181 - accuracy: 0.9659
1591/1688 [===========================>..] - ETA: 0s - loss: 0.6154 - accuracy: 0.9660
1610/1688 [===========================>..] - ETA: 0s - loss: 0.6153 - accuracy: 0.9660
1630/1688 [===========================>..] - ETA: 0s - loss: 0.6158 - accuracy: 0.9660
1649/1688 [============================>.] - ETA: 0s - loss: 0.6134 - accuracy: 0.9660
1668/1688 [============================>.] - ETA: 0s - loss: 0.6165 - accuracy: 0.9658
1687/1688 [============================>.] - ETA: 0s - loss: 0.6171 - accuracy: 0.9657
1688/1688 [==============================] - 5s 3ms/step - loss: 0.6169 - accuracy: 0.9658 - val_loss: 0.7428 - val_accuracy: 0.9697
Epoch 3/5
1/1688 [..............................] - ETA: 4s - loss: 0.0610 - accuracy: 0.9688
21/1688 [..............................] - ETA: 4s - loss: 0.4194 - accuracy: 0.9821
40/1688 [..............................] - ETA: 4s - loss: 0.4001 - accuracy: 0.9781
59/1688 [>.............................] - ETA: 4s - loss: 0.3958 - accuracy: 0.9793
79/1688 [>.............................] - ETA: 4s - loss: 0.3809 - accuracy: 0.9794
98/1688 [>.............................] - ETA: 4s - loss: 0.4410 - accuracy: 0.9780
117/1688 [=>............................] - ETA: 4s - loss: 0.4433 - accuracy: 0.9768
136/1688 [=>............................] - ETA: 4s - loss: 0.4718 - accuracy: 0.9761
155/1688 [=>............................] - ETA: 4s - loss: 0.4682 - accuracy: 0.9758
174/1688 [==>...........................] - ETA: 4s - loss: 0.4840 - accuracy: 0.9749
194/1688 [==>...........................] - ETA: 3s - loss: 0.5004 - accuracy: 0.9744
213/1688 [==>...........................] - ETA: 3s - loss: 0.4965 - accuracy: 0.9745
232/1688 [===>..........................] - ETA: 3s - loss: 0.4960 - accuracy: 0.9744
252/1688 [===>..........................] - ETA: 3s - loss: 0.4993 - accuracy: 0.9747
271/1688 [===>..........................] - ETA: 3s - loss: 0.5006 - accuracy: 0.9744
291/1688 [====>.........................] - ETA: 3s - loss: 0.5046 - accuracy: 0.9745
310/1688 [====>.........................] - ETA: 3s - loss: 0.5078 - accuracy: 0.9740
330/1688 [====>.........................] - ETA: 3s - loss: 0.5027 - accuracy: 0.9741
349/1688 [=====>........................] - ETA: 3s - loss: 0.5087 - accuracy: 0.9743
369/1688 [=====>........................] - ETA: 3s - loss: 0.5215 - accuracy: 0.9741
388/1688 [=====>........................] - ETA: 3s - loss: 0.5345 - accuracy: 0.9739
407/1688 [======>.......................] - ETA: 3s - loss: 0.5380 - accuracy: 0.9740
427/1688 [======>.......................] - ETA: 3s - loss: 0.5405 - accuracy: 0.9737
446/1688 [======>.......................] - ETA: 3s - loss: 0.5497 - accuracy: 0.9734
465/1688 [=======>......................] - ETA: 3s - loss: 0.5547 - accuracy: 0.9733
485/1688 [=======>......................] - ETA: 3s - loss: 0.5672 - accuracy: 0.9727
504/1688 [=======>......................] - ETA: 3s - loss: 0.5662 - accuracy: 0.9724
523/1688 [========>.....................] - ETA: 3s - loss: 0.5708 - accuracy: 0.9720
543/1688 [========>.....................] - ETA: 3s - loss: 0.5767 - accuracy: 0.9715
563/1688 [=========>....................] - ETA: 2s - loss: 0.5754 - accuracy: 0.9716
583/1688 [=========>....................] - ETA: 2s - loss: 0.5738 - accuracy: 0.9716
602/1688 [=========>....................] - ETA: 2s - loss: 0.5608 - accuracy: 0.9721
621/1688 [==========>...................] - ETA: 2s - loss: 0.5618 - accuracy: 0.9721
641/1688 [==========>...................] - ETA: 2s - loss: 0.5697 - accuracy: 0.9715
660/1688 [==========>...................] - ETA: 2s - loss: 0.5678 - accuracy: 0.9716
679/1688 [===========>..................] - ETA: 2s - loss: 0.5632 - accuracy: 0.9716
699/1688 [===========>..................] - ETA: 2s - loss: 0.5636 - accuracy: 0.9717
718/1688 [===========>..................] - ETA: 2s - loss: 0.5615 - accuracy: 0.9718
737/1688 [============>.................] - ETA: 2s - loss: 0.5610 - accuracy: 0.9718
756/1688 [============>.................] - ETA: 2s - loss: 0.5674 - accuracy: 0.9717
775/1688 [============>.................] - ETA: 2s - loss: 0.5624 - accuracy: 0.9718
795/1688 [=============>................] - ETA: 2s - loss: 0.5597 - accuracy: 0.9719
815/1688 [=============>................] - ETA: 2s - loss: 0.5603 - accuracy: 0.9719
835/1688 [=============>................] - ETA: 2s - loss: 0.5558 - accuracy: 0.9722
854/1688 [==============>...............] - ETA: 2s - loss: 0.5523 - accuracy: 0.9722
873/1688 [==============>...............] - ETA: 2s - loss: 0.5536 - accuracy: 0.9720
892/1688 [==============>...............] - ETA: 2s - loss: 0.5555 - accuracy: 0.9719
912/1688 [===============>..............] - ETA: 2s - loss: 0.5687 - accuracy: 0.9715
932/1688 [===============>..............] - ETA: 1s - loss: 0.5692 - accuracy: 0.9717
951/1688 [===============>..............] - ETA: 1s - loss: 0.5686 - accuracy: 0.9717
970/1688 [================>.............] - ETA: 1s - loss: 0.5673 - accuracy: 0.9718
989/1688 [================>.............] - ETA: 1s - loss: 0.5619 - accuracy: 0.9720
1008/1688 [================>.............] - ETA: 1s - loss: 0.5547 - accuracy: 0.9721
1028/1688 [=================>............] - ETA: 1s - loss: 0.5534 - accuracy: 0.9722
1048/1688 [=================>............] - ETA: 1s - loss: 0.5574 - accuracy: 0.9722
1068/1688 [=================>............] - ETA: 1s - loss: 0.5615 - accuracy: 0.9722
1087/1688 [==================>...........] - ETA: 1s - loss: 0.5607 - accuracy: 0.9723
1106/1688 [==================>...........] - ETA: 1s - loss: 0.5628 - accuracy: 0.9723
1125/1688 [==================>...........] - ETA: 1s - loss: 0.5643 - accuracy: 0.9723
1145/1688 [===================>..........] - ETA: 1s - loss: 0.5631 - accuracy: 0.9723
1164/1688 [===================>..........] - ETA: 1s - loss: 0.5631 - accuracy: 0.9723
1183/1688 [====================>.........] - ETA: 1s - loss: 0.5626 - accuracy: 0.9722
1202/1688 [====================>.........] - ETA: 1s - loss: 0.5622 - accuracy: 0.9723
1221/1688 [====================>.........] - ETA: 1s - loss: 0.5636 - accuracy: 0.9722
1241/1688 [=====================>........] - ETA: 1s - loss: 0.5672 - accuracy: 0.9721
1260/1688 [=====================>........] - ETA: 1s - loss: 0.5701 - accuracy: 0.9721
1279/1688 [=====================>........] - ETA: 1s - loss: 0.5699 - accuracy: 0.9720
1299/1688 [======================>.......] - ETA: 1s - loss: 0.5736 - accuracy: 0.9719
1319/1688 [======================>.......] - ETA: 0s - loss: 0.5748 - accuracy: 0.9718
1338/1688 [======================>.......] - ETA: 0s - loss: 0.5714 - accuracy: 0.9720
1357/1688 [=======================>......] - ETA: 0s - loss: 0.5713 - accuracy: 0.9719
1376/1688 [=======================>......] - ETA: 0s - loss: 0.5750 - accuracy: 0.9718
1395/1688 [=======================>......] - ETA: 0s - loss: 0.5749 - accuracy: 0.9718
1415/1688 [========================>.....] - ETA: 0s - loss: 0.5736 - accuracy: 0.9719
1434/1688 [========================>.....] - ETA: 0s - loss: 0.5733 - accuracy: 0.9718
1454/1688 [========================>.....] - ETA: 0s - loss: 0.5737 - accuracy: 0.9718
1474/1688 [=========================>....] - ETA: 0s - loss: 0.5767 - accuracy: 0.9717
1494/1688 [=========================>....] - ETA: 0s - loss: 0.5789 - accuracy: 0.9716
1513/1688 [=========================>....] - ETA: 0s - loss: 0.5801 - accuracy: 0.9716
1533/1688 [==========================>...] - ETA: 0s - loss: 0.5817 - accuracy: 0.9715
1552/1688 [==========================>...] - ETA: 0s - loss: 0.5823 - accuracy: 0.9715
1571/1688 [==========================>...] - ETA: 0s - loss: 0.5807 - accuracy: 0.9715
1590/1688 [===========================>..] - ETA: 0s - loss: 0.5780 - accuracy: 0.9716
1609/1688 [===========================>..] - ETA: 0s - loss: 0.5772 - accuracy: 0.9716
1629/1688 [===========================>..] - ETA: 0s - loss: 0.5791 - accuracy: 0.9716
1648/1688 [============================>.] - ETA: 0s - loss: 0.5765 - accuracy: 0.9716
1667/1688 [============================>.] - ETA: 0s - loss: 0.5750 - accuracy: 0.9716
1686/1688 [============================>.] - ETA: 0s - loss: 0.5746 - accuracy: 0.9717
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5742 - accuracy: 0.9717 - val_loss: 0.5708 - val_accuracy: 0.9758
Epoch 4/5
1/1688 [..............................] - ETA: 4s - loss: 0.0000e+00 - accuracy: 1.0000
21/1688 [..............................] - ETA: 4s - loss: 0.4838 - accuracy: 0.9777
41/1688 [..............................] - ETA: 4s - loss: 0.4347 - accuracy: 0.9809
60/1688 [>.............................] - ETA: 4s - loss: 0.4119 - accuracy: 0.9823
79/1688 [>.............................] - ETA: 4s - loss: 0.4530 - accuracy: 0.9798
98/1688 [>.............................] - ETA: 4s - loss: 0.4212 - accuracy: 0.9809
118/1688 [=>............................] - ETA: 4s - loss: 0.3986 - accuracy: 0.9823
137/1688 [=>............................] - ETA: 4s - loss: 0.4119 - accuracy: 0.9808
156/1688 [=>............................] - ETA: 4s - loss: 0.4180 - accuracy: 0.9804
175/1688 [==>...........................] - ETA: 3s - loss: 0.4470 - accuracy: 0.9786
194/1688 [==>...........................] - ETA: 3s - loss: 0.4437 - accuracy: 0.9784
214/1688 [==>...........................] - ETA: 3s - loss: 0.4457 - accuracy: 0.9777
234/1688 [===>..........................] - ETA: 3s - loss: 0.4454 - accuracy: 0.9770
254/1688 [===>..........................] - ETA: 3s - loss: 0.4382 - accuracy: 0.9771
273/1688 [===>..........................] - ETA: 3s - loss: 0.4361 - accuracy: 0.9772
293/1688 [====>.........................] - ETA: 3s - loss: 0.4203 - accuracy: 0.9779
313/1688 [====>.........................] - ETA: 3s - loss: 0.4149 - accuracy: 0.9779
333/1688 [====>.........................] - ETA: 3s - loss: 0.4093 - accuracy: 0.9782
352/1688 [=====>........................] - ETA: 3s - loss: 0.4170 - accuracy: 0.9782
372/1688 [=====>........................] - ETA: 3s - loss: 0.4183 - accuracy: 0.9782
391/1688 [=====>........................] - ETA: 3s - loss: 0.4169 - accuracy: 0.9783
410/1688 [======>.......................] - ETA: 3s - loss: 0.4135 - accuracy: 0.9784
429/1688 [======>.......................] - ETA: 3s - loss: 0.4306 - accuracy: 0.9779
448/1688 [======>.......................] - ETA: 3s - loss: 0.4261 - accuracy: 0.9779
467/1688 [=======>......................] - ETA: 3s - loss: 0.4243 - accuracy: 0.9779
486/1688 [=======>......................] - ETA: 3s - loss: 0.4283 - accuracy: 0.9775
505/1688 [=======>......................] - ETA: 3s - loss: 0.4340 - accuracy: 0.9770
524/1688 [========>.....................] - ETA: 3s - loss: 0.4287 - accuracy: 0.9772
543/1688 [========>.....................] - ETA: 3s - loss: 0.4412 - accuracy: 0.9770
563/1688 [=========>....................] - ETA: 2s - loss: 0.4412 - accuracy: 0.9766
582/1688 [=========>....................] - ETA: 2s - loss: 0.4462 - accuracy: 0.9764
601/1688 [=========>....................] - ETA: 2s - loss: 0.4470 - accuracy: 0.9761
621/1688 [==========>...................] - ETA: 2s - loss: 0.4457 - accuracy: 0.9760
640/1688 [==========>...................] - ETA: 2s - loss: 0.4432 - accuracy: 0.9760
659/1688 [==========>...................] - ETA: 2s - loss: 0.4473 - accuracy: 0.9757
678/1688 [===========>..................] - ETA: 2s - loss: 0.4451 - accuracy: 0.9758
697/1688 [===========>..................] - ETA: 2s - loss: 0.4488 - accuracy: 0.9758
717/1688 [===========>..................] - ETA: 2s - loss: 0.4546 - accuracy: 0.9756
736/1688 [============>.................] - ETA: 2s - loss: 0.4567 - accuracy: 0.9756
755/1688 [============>.................] - ETA: 2s - loss: 0.4561 - accuracy: 0.9755
774/1688 [============>.................] - ETA: 2s - loss: 0.4597 - accuracy: 0.9753
794/1688 [=============>................] - ETA: 2s - loss: 0.4590 - accuracy: 0.9753
813/1688 [=============>................] - ETA: 2s - loss: 0.4598 - accuracy: 0.9751
833/1688 [=============>................] - ETA: 2s - loss: 0.4627 - accuracy: 0.9752
852/1688 [==============>...............] - ETA: 2s - loss: 0.4589 - accuracy: 0.9755
871/1688 [==============>...............] - ETA: 2s - loss: 0.4652 - accuracy: 0.9754
890/1688 [==============>...............] - ETA: 2s - loss: 0.4633 - accuracy: 0.9754
909/1688 [===============>..............] - ETA: 2s - loss: 0.4650 - accuracy: 0.9752
928/1688 [===============>..............] - ETA: 2s - loss: 0.4617 - accuracy: 0.9753
947/1688 [===============>..............] - ETA: 1s - loss: 0.4609 - accuracy: 0.9754
966/1688 [================>.............] - ETA: 1s - loss: 0.4627 - accuracy: 0.9754
985/1688 [================>.............] - ETA: 1s - loss: 0.4639 - accuracy: 0.9754
1004/1688 [================>.............] - ETA: 1s - loss: 0.4600 - accuracy: 0.9755
1023/1688 [=================>............] - ETA: 1s - loss: 0.4569 - accuracy: 0.9755
1043/1688 [=================>............] - ETA: 1s - loss: 0.4535 - accuracy: 0.9755
1062/1688 [=================>............] - ETA: 1s - loss: 0.4490 - accuracy: 0.9756
1082/1688 [==================>...........] - ETA: 1s - loss: 0.4563 - accuracy: 0.9755
1101/1688 [==================>...........] - ETA: 1s - loss: 0.4524 - accuracy: 0.9756
1121/1688 [==================>...........] - ETA: 1s - loss: 0.4545 - accuracy: 0.9756
1141/1688 [===================>..........] - ETA: 1s - loss: 0.4601 - accuracy: 0.9755
1161/1688 [===================>..........] - ETA: 1s - loss: 0.4602 - accuracy: 0.9754
1181/1688 [===================>..........] - ETA: 1s - loss: 0.4586 - accuracy: 0.9754
1200/1688 [====================>.........] - ETA: 1s - loss: 0.4599 - accuracy: 0.9751
1219/1688 [====================>.........] - ETA: 1s - loss: 0.4574 - accuracy: 0.9752
1238/1688 [=====================>........] - ETA: 1s - loss: 0.4564 - accuracy: 0.9753
1258/1688 [=====================>........] - ETA: 1s - loss: 0.4604 - accuracy: 0.9753
1278/1688 [=====================>........] - ETA: 1s - loss: 0.4585 - accuracy: 0.9753
1297/1688 [======================>.......] - ETA: 1s - loss: 0.4609 - accuracy: 0.9753
1316/1688 [======================>.......] - ETA: 0s - loss: 0.4587 - accuracy: 0.9754
1335/1688 [======================>.......] - ETA: 0s - loss: 0.4637 - accuracy: 0.9753
1354/1688 [=======================>......] - ETA: 0s - loss: 0.4603 - accuracy: 0.9754
1373/1688 [=======================>......] - ETA: 0s - loss: 0.4601 - accuracy: 0.9754
1392/1688 [=======================>......] - ETA: 0s - loss: 0.4622 - accuracy: 0.9754
1411/1688 [========================>.....] - ETA: 0s - loss: 0.4617 - accuracy: 0.9753
1430/1688 [========================>.....] - ETA: 0s - loss: 0.4645 - accuracy: 0.9752
1449/1688 [========================>.....] - ETA: 0s - loss: 0.4664 - accuracy: 0.9752
1468/1688 [=========================>....] - ETA: 0s - loss: 0.4655 - accuracy: 0.9753
1487/1688 [=========================>....] - ETA: 0s - loss: 0.4679 - accuracy: 0.9753
1506/1688 [=========================>....] - ETA: 0s - loss: 0.4692 - accuracy: 0.9753
1525/1688 [==========================>...] - ETA: 0s - loss: 0.4720 - accuracy: 0.9753
1544/1688 [==========================>...] - ETA: 0s - loss: 0.4732 - accuracy: 0.9752
1564/1688 [==========================>...] - ETA: 0s - loss: 0.4720 - accuracy: 0.9752
1584/1688 [===========================>..] - ETA: 0s - loss: 0.4736 - accuracy: 0.9752
1603/1688 [===========================>..] - ETA: 0s - loss: 0.4732 - accuracy: 0.9753
1622/1688 [===========================>..] - ETA: 0s - loss: 0.4762 - accuracy: 0.9753
1641/1688 [============================>.] - ETA: 0s - loss: 0.4745 - accuracy: 0.9753
1661/1688 [============================>.] - ETA: 0s - loss: 0.4756 - accuracy: 0.9753
1680/1688 [============================>.] - ETA: 0s - loss: 0.4761 - accuracy: 0.9752
1688/1688 [==============================] - 5s 3ms/step - loss: 0.4756 - accuracy: 0.9752 - val_loss: 0.6931 - val_accuracy: 0.9790
Epoch 5/5
1/1688 [..............................] - ETA: 4s - loss: 0.3281 - accuracy: 0.9688
21/1688 [..............................] - ETA: 4s - loss: 0.3235 - accuracy: 0.9777
41/1688 [..............................] - ETA: 4s - loss: 0.2983 - accuracy: 0.9825
61/1688 [>.............................] - ETA: 4s - loss: 0.2984 - accuracy: 0.9826
80/1688 [>.............................] - ETA: 4s - loss: 0.2996 - accuracy: 0.9820
100/1688 [>.............................] - ETA: 4s - loss: 0.3342 - accuracy: 0.9809
119/1688 [=>............................] - ETA: 4s - loss: 0.3210 - accuracy: 0.9819
139/1688 [=>............................] - ETA: 4s - loss: 0.3913 - accuracy: 0.9800
159/1688 [=>............................] - ETA: 3s - loss: 0.3805 - accuracy: 0.9800
178/1688 [==>...........................] - ETA: 3s - loss: 0.3871 - accuracy: 0.9789
197/1688 [==>...........................] - ETA: 3s - loss: 0.4139 - accuracy: 0.9784
216/1688 [==>...........................] - ETA: 3s - loss: 0.4324 - accuracy: 0.9782
235/1688 [===>..........................] - ETA: 3s - loss: 0.4209 - accuracy: 0.9782
254/1688 [===>..........................] - ETA: 3s - loss: 0.4272 - accuracy: 0.9779
274/1688 [===>..........................] - ETA: 3s - loss: 0.4277 - accuracy: 0.9775
293/1688 [====>.........................] - ETA: 3s - loss: 0.4137 - accuracy: 0.9778
313/1688 [====>.........................] - ETA: 3s - loss: 0.4051 - accuracy: 0.9780
332/1688 [====>.........................] - ETA: 3s - loss: 0.3927 - accuracy: 0.9786
351/1688 [=====>........................] - ETA: 3s - loss: 0.3970 - accuracy: 0.9786
370/1688 [=====>........................] - ETA: 3s - loss: 0.3941 - accuracy: 0.9786
390/1688 [=====>........................] - ETA: 3s - loss: 0.4063 - accuracy: 0.9780
409/1688 [======>.......................] - ETA: 3s - loss: 0.4046 - accuracy: 0.9781
428/1688 [======>.......................] - ETA: 3s - loss: 0.4055 - accuracy: 0.9779
447/1688 [======>.......................] - ETA: 3s - loss: 0.4078 - accuracy: 0.9779
466/1688 [=======>......................] - ETA: 3s - loss: 0.4022 - accuracy: 0.9782
486/1688 [=======>......................] - ETA: 3s - loss: 0.3984 - accuracy: 0.9782
505/1688 [=======>......................] - ETA: 3s - loss: 0.3963 - accuracy: 0.9787
524/1688 [========>.....................] - ETA: 3s - loss: 0.3971 - accuracy: 0.9786
544/1688 [========>.....................] - ETA: 3s - loss: 0.3953 - accuracy: 0.9787
563/1688 [=========>....................] - ETA: 2s - loss: 0.3971 - accuracy: 0.9785
582/1688 [=========>....................] - ETA: 2s - loss: 0.3969 - accuracy: 0.9785
601/1688 [=========>....................] - ETA: 2s - loss: 0.3996 - accuracy: 0.9785
620/1688 [==========>...................] - ETA: 2s - loss: 0.4068 - accuracy: 0.9783
640/1688 [==========>...................] - ETA: 2s - loss: 0.4105 - accuracy: 0.9781
659/1688 [==========>...................] - ETA: 2s - loss: 0.4221 - accuracy: 0.9777
679/1688 [===========>..................] - ETA: 2s - loss: 0.4200 - accuracy: 0.9778
698/1688 [===========>..................] - ETA: 2s - loss: 0.4214 - accuracy: 0.9778
717/1688 [===========>..................] - ETA: 2s - loss: 0.4296 - accuracy: 0.9775
737/1688 [============>.................] - ETA: 2s - loss: 0.4331 - accuracy: 0.9774
756/1688 [============>.................] - ETA: 2s - loss: 0.4349 - accuracy: 0.9775
774/1688 [============>.................] - ETA: 2s - loss: 0.4360 - accuracy: 0.9775
792/1688 [=============>................] - ETA: 2s - loss: 0.4454 - accuracy: 0.9774
810/1688 [=============>................] - ETA: 2s - loss: 0.4407 - accuracy: 0.9776
828/1688 [=============>................] - ETA: 2s - loss: 0.4467 - accuracy: 0.9774
846/1688 [==============>...............] - ETA: 2s - loss: 0.4483 - accuracy: 0.9774
865/1688 [==============>...............] - ETA: 2s - loss: 0.4550 - accuracy: 0.9770
883/1688 [==============>...............] - ETA: 2s - loss: 0.4565 - accuracy: 0.9770
901/1688 [===============>..............] - ETA: 2s - loss: 0.4549 - accuracy: 0.9771
919/1688 [===============>..............] - ETA: 2s - loss: 0.4506 - accuracy: 0.9772
937/1688 [===============>..............] - ETA: 2s - loss: 0.4561 - accuracy: 0.9771
955/1688 [===============>..............] - ETA: 1s - loss: 0.4622 - accuracy: 0.9769
973/1688 [================>.............] - ETA: 1s - loss: 0.4630 - accuracy: 0.9767
991/1688 [================>.............] - ETA: 1s - loss: 0.4600 - accuracy: 0.9768
1010/1688 [================>.............] - ETA: 1s - loss: 0.4602 - accuracy: 0.9767
1029/1688 [=================>............] - ETA: 1s - loss: 0.4655 - accuracy: 0.9765
1049/1688 [=================>............] - ETA: 1s - loss: 0.4658 - accuracy: 0.9763
1068/1688 [=================>............] - ETA: 1s - loss: 0.4664 - accuracy: 0.9763
1087/1688 [==================>...........] - ETA: 1s - loss: 0.4651 - accuracy: 0.9763
1106/1688 [==================>...........] - ETA: 1s - loss: 0.4612 - accuracy: 0.9764
1126/1688 [===================>..........] - ETA: 1s - loss: 0.4663 - accuracy: 0.9763
1146/1688 [===================>..........] - ETA: 1s - loss: 0.4645 - accuracy: 0.9765
1165/1688 [===================>..........] - ETA: 1s - loss: 0.4628 - accuracy: 0.9766
1184/1688 [====================>.........] - ETA: 1s - loss: 0.4621 - accuracy: 0.9766
1203/1688 [====================>.........] - ETA: 1s - loss: 0.4619 - accuracy: 0.9766
1223/1688 [====================>.........] - ETA: 1s - loss: 0.4614 - accuracy: 0.9766
1242/1688 [=====================>........] - ETA: 1s - loss: 0.4615 - accuracy: 0.9766
1261/1688 [=====================>........] - ETA: 1s - loss: 0.4604 - accuracy: 0.9766
1280/1688 [=====================>........] - ETA: 1s - loss: 0.4610 - accuracy: 0.9766
1299/1688 [======================>.......] - ETA: 1s - loss: 0.4632 - accuracy: 0.9766
1318/1688 [======================>.......] - ETA: 0s - loss: 0.4621 - accuracy: 0.9767
1337/1688 [======================>.......] - ETA: 0s - loss: 0.4649 - accuracy: 0.9766
1356/1688 [=======================>......] - ETA: 0s - loss: 0.4627 - accuracy: 0.9767
1375/1688 [=======================>......] - ETA: 0s - loss: 0.4601 - accuracy: 0.9768
1394/1688 [=======================>......] - ETA: 0s - loss: 0.4630 - accuracy: 0.9766
1414/1688 [========================>.....] - ETA: 0s - loss: 0.4629 - accuracy: 0.9767
1433/1688 [========================>.....] - ETA: 0s - loss: 0.4665 - accuracy: 0.9765
1452/1688 [========================>.....] - ETA: 0s - loss: 0.4681 - accuracy: 0.9765
1471/1688 [=========================>....] - ETA: 0s - loss: 0.4732 - accuracy: 0.9764
1491/1688 [=========================>....] - ETA: 0s - loss: 0.4713 - accuracy: 0.9765
1510/1688 [=========================>....] - ETA: 0s - loss: 0.4724 - accuracy: 0.9765
1529/1688 [==========================>...] - ETA: 0s - loss: 0.4708 - accuracy: 0.9765
1549/1688 [==========================>...] - ETA: 0s - loss: 0.4701 - accuracy: 0.9766
1569/1688 [==========================>...] - ETA: 0s - loss: 0.4704 - accuracy: 0.9766
1588/1688 [===========================>..] - ETA: 0s - loss: 0.4699 - accuracy: 0.9766
1607/1688 [===========================>..] - ETA: 0s - loss: 0.4709 - accuracy: 0.9767
1626/1688 [===========================>..] - ETA: 0s - loss: 0.4704 - accuracy: 0.9766
1645/1688 [============================>.] - ETA: 0s - loss: 0.4691 - accuracy: 0.9767
1664/1688 [============================>.] - ETA: 0s - loss: 0.4672 - accuracy: 0.9767
1683/1688 [============================>.] - ETA: 0s - loss: 0.4689 - accuracy: 0.9765
1688/1688 [==============================] - 5s 3ms/step - loss: 0.4683 - accuracy: 0.9766 - val_loss: 0.6237 - val_accuracy: 0.9738
Test accuracy after fine tuning: 0.9715999960899353
6. Model conversion
After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida software simulator.
Note
One needs to supply the coefficients used to rescale the input
dataset before the training - here input_scaling.
As with Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()
accuracy = model_akida.evaluate(raw_x_test, raw_y_test)
print('Test accuracy after conversion:', accuracy)
# For non-regression purpose
assert accuracy > 0.96
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 3
______________________________________________
______________________________________________________________
Layer (type) Output shape Kernel shape
================= SW/conv2d-dense (Software) =================
conv2d (InputConv.) [13, 13, 32] (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.) [7, 7, 64] (3, 3, 32, 1)
______________________________________________________________
(1, 1, 32, 64)
______________________________________________________________
dense (Fully.) [1, 1, 10] (1, 1, 3136, 10)
______________________________________________________________
Test accuracy after conversion: 0.9696000218391418
Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).
Total running time of the script: ( 1 minutes 2.268 seconds)