Global Akida workflow

Using the MNIST dataset, this example shows the definition and training of a keras floating point model, its quantization to 8-bit with the help of calibration, its quantization to 4-bit using QAT and its conversion to Akida. Notice that the performance of the original keras floating point model is maintained throughout the Akida flow. Please refer to the Akida user guide for further information.

Note

Please refer to the TensorFlow tf.keras.models module for model creation/import details and the TensorFlow Guide for TensorFlow usage.

The MNIST example below is light enough so that a GPU is not needed for training.

Global Akida workflow

1. Create and train

1.1. Load and reshape MNIST dataset

import numpy as np

import matplotlib.cm as cm
import matplotlib.pyplot as plt

from keras.datasets import mnist

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Add a channels dimension to the image sets as Akida expects 4-D inputs (corresponding to
# (num_samples, width, height, channels). Note: MNIST is a grayscale dataset and is unusual
# in this respect - most image data already includes a channel dimension, and this step will
# not be necessary.
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)

# Display a few images from the test set
f, axarr = plt.subplots(1, 4)
for i in range(0, 4):
    axarr[i].imshow(x_test[i].reshape((28, 28)), cmap=cm.Greys_r)
    axarr[i].set_title('Class %d' % y_test[i])
plt.show()

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz

    8192/11490434 [..............................] - ETA: 0s
   16384/11490434 [..............................] - ETA: 1:37
   32768/11490434 [..............................] - ETA: 1:09
   49152/11490434 [..............................] - ETA: 1:00
   65536/11490434 [..............................] - ETA: 56s 
   90112/11490434 [..............................] - ETA: 49s
  114688/11490434 [..............................] - ETA: 44s
  155648/11490434 [..............................] - ETA: 36s
  196608/11490434 [..............................] - ETA: 32s
  245760/11490434 [..............................] - ETA: 28s
  311296/11490434 [..............................] - ETA: 23s
  393216/11490434 [>.............................] - ETA: 20s
  499712/11490434 [>.............................] - ETA: 16s
  622592/11490434 [>.............................] - ETA: 14s
  778240/11490434 [=>............................] - ETA: 12s
  974848/11490434 [=>............................] - ETA: 9s 
 1212416/11490434 [==>...........................] - ETA: 8s
 1540096/11490434 [===>..........................] - ETA: 6s
 1941504/11490434 [====>.........................] - ETA: 5s
 2433024/11490434 [=====>........................] - ETA: 4s
 3047424/11490434 [======>.......................] - ETA: 3s
 3768320/11490434 [========>.....................] - ETA: 2s
 4849664/11490434 [===========>..................] - ETA: 1s
 6078464/11490434 [==============>...............] - ETA: 1s
 6848512/11490434 [================>.............] - ETA: 0s
 7995392/11490434 [===================>..........] - ETA: 0s
 9224192/11490434 [=======================>......] - ETA: 0s
 9994240/11490434 [=========================>....] - ETA: 0s
11132928/11490434 [============================>.] - ETA: 0s
11490434/11490434 [==============================] - 2s 0us/step

1.2. Model definition

Note that at this stage, there is nothing specific to the Akida IP. The model constructed below, as inspired by this example, is a completely standard Keras CNN model.

import keras

model_keras = keras.models.Sequential([
    keras.layers.Rescaling(1. / 255, input_shape=(28, 28, 1)),
    keras.layers.Conv2D(filters=32, kernel_size=3, strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    # Separable layer
    keras.layers.DepthwiseConv2D(kernel_size=3, padding='same', strides=2),
    keras.layers.Conv2D(filters=64, kernel_size=1, padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()

Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling (Rescaling)       (None, 28, 28, 1)         0

 conv2d (Conv2D)             (None, 13, 13, 32)        320

 batch_normalization (BatchN  (None, 13, 13, 32)       128
 ormalization)

 re_lu (ReLU)                (None, 13, 13, 32)        0

 depthwise_conv2d (Depthwise  (None, 7, 7, 32)         320
 Conv2D)

 conv2d_1 (Conv2D)           (None, 7, 7, 64)          2112

 batch_normalization_1 (Batc  (None, 7, 7, 64)         256
 hNormalization)

 re_lu_1 (ReLU)              (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (Dense)               (None, 10)                31370

=================================================================
Total params: 34,506
Trainable params: 34,314
Non-trainable params: 192
_________________________________________________________________

1.3. Model training

Given the model created above, train the model and check its accuracy. The model should achieve a test accuracy over 98% after 10 epochs.

from keras.optimizers import Adam

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-3),
    metrics=['accuracy'])

_ = model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)

Epoch 1/10

   1/1688 [..............................] - ETA: 21:03 - loss: 2.8114 - accuracy: 0.0312
  30/1688 [..............................] - ETA: 2s - loss: 1.2280 - accuracy: 0.5990   
  60/1688 [>.............................] - ETA: 2s - loss: 0.8640 - accuracy: 0.7240
  89/1688 [>.............................] - ETA: 2s - loss: 0.7164 - accuracy: 0.7770
 119/1688 [=>............................] - ETA: 2s - loss: 0.6162 - accuracy: 0.8101
 149/1688 [=>............................] - ETA: 2s - loss: 0.5643 - accuracy: 0.8276
 179/1688 [==>...........................] - ETA: 2s - loss: 0.5166 - accuracy: 0.8434
 208/1688 [==>...........................] - ETA: 2s - loss: 0.4771 - accuracy: 0.8558
 238/1688 [===>..........................] - ETA: 2s - loss: 0.4449 - accuracy: 0.8650
 268/1688 [===>..........................] - ETA: 2s - loss: 0.4196 - accuracy: 0.8731
 298/1688 [====>.........................] - ETA: 2s - loss: 0.4033 - accuracy: 0.8786
 328/1688 [====>.........................] - ETA: 2s - loss: 0.3854 - accuracy: 0.8839
 359/1688 [=====>........................] - ETA: 2s - loss: 0.3668 - accuracy: 0.8908
 390/1688 [=====>........................] - ETA: 2s - loss: 0.3510 - accuracy: 0.8961
 420/1688 [======>.......................] - ETA: 2s - loss: 0.3383 - accuracy: 0.8996
 450/1688 [======>.......................] - ETA: 2s - loss: 0.3275 - accuracy: 0.9031
 481/1688 [=======>......................] - ETA: 2s - loss: 0.3182 - accuracy: 0.9058
 511/1688 [========>.....................] - ETA: 1s - loss: 0.3114 - accuracy: 0.9077
 540/1688 [========>.....................] - ETA: 1s - loss: 0.3043 - accuracy: 0.9102
 570/1688 [=========>....................] - ETA: 1s - loss: 0.2971 - accuracy: 0.9123
 600/1688 [=========>....................] - ETA: 1s - loss: 0.2881 - accuracy: 0.9151
 630/1688 [==========>...................] - ETA: 1s - loss: 0.2811 - accuracy: 0.9172
 660/1688 [==========>...................] - ETA: 1s - loss: 0.2734 - accuracy: 0.9195
 690/1688 [===========>..................] - ETA: 1s - loss: 0.2666 - accuracy: 0.9215
 720/1688 [===========>..................] - ETA: 1s - loss: 0.2605 - accuracy: 0.9234
 750/1688 [============>.................] - ETA: 1s - loss: 0.2555 - accuracy: 0.9249
 781/1688 [============>.................] - ETA: 1s - loss: 0.2512 - accuracy: 0.9263
 812/1688 [=============>................] - ETA: 1s - loss: 0.2463 - accuracy: 0.9279
 843/1688 [=============>................] - ETA: 1s - loss: 0.2421 - accuracy: 0.9288
 874/1688 [==============>...............] - ETA: 1s - loss: 0.2371 - accuracy: 0.9303
 904/1688 [===============>..............] - ETA: 1s - loss: 0.2333 - accuracy: 0.9315
 935/1688 [===============>..............] - ETA: 1s - loss: 0.2302 - accuracy: 0.9325
 965/1688 [================>.............] - ETA: 1s - loss: 0.2273 - accuracy: 0.9334
 995/1688 [================>.............] - ETA: 1s - loss: 0.2247 - accuracy: 0.9342
1025/1688 [=================>............] - ETA: 1s - loss: 0.2216 - accuracy: 0.9350
1055/1688 [=================>............] - ETA: 1s - loss: 0.2178 - accuracy: 0.9360
1086/1688 [==================>...........] - ETA: 1s - loss: 0.2151 - accuracy: 0.9368
1117/1688 [==================>...........] - ETA: 0s - loss: 0.2116 - accuracy: 0.9377
1148/1688 [===================>..........] - ETA: 0s - loss: 0.2084 - accuracy: 0.9385
1178/1688 [===================>..........] - ETA: 0s - loss: 0.2055 - accuracy: 0.9394
1208/1688 [====================>.........] - ETA: 0s - loss: 0.2031 - accuracy: 0.9400
1239/1688 [=====================>........] - ETA: 0s - loss: 0.2010 - accuracy: 0.9406
1269/1688 [=====================>........] - ETA: 0s - loss: 0.1987 - accuracy: 0.9412
1299/1688 [======================>.......] - ETA: 0s - loss: 0.1974 - accuracy: 0.9416
1329/1688 [======================>.......] - ETA: 0s - loss: 0.1950 - accuracy: 0.9424
1359/1688 [=======================>......] - ETA: 0s - loss: 0.1922 - accuracy: 0.9432
1389/1688 [=======================>......] - ETA: 0s - loss: 0.1909 - accuracy: 0.9436
1419/1688 [========================>.....] - ETA: 0s - loss: 0.1890 - accuracy: 0.9441
1449/1688 [========================>.....] - ETA: 0s - loss: 0.1869 - accuracy: 0.9447
1480/1688 [=========================>....] - ETA: 0s - loss: 0.1844 - accuracy: 0.9456
1510/1688 [=========================>....] - ETA: 0s - loss: 0.1826 - accuracy: 0.9459
1541/1688 [==========================>...] - ETA: 0s - loss: 0.1809 - accuracy: 0.9464
1571/1688 [==========================>...] - ETA: 0s - loss: 0.1794 - accuracy: 0.9468
1602/1688 [===========================>..] - ETA: 0s - loss: 0.1776 - accuracy: 0.9474
1632/1688 [============================>.] - ETA: 0s - loss: 0.1760 - accuracy: 0.9479
1662/1688 [============================>.] - ETA: 0s - loss: 0.1741 - accuracy: 0.9485
1688/1688 [==============================] - ETA: 0s - loss: 0.1725 - accuracy: 0.9489
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1725 - accuracy: 0.9489 - val_loss: 0.0720 - val_accuracy: 0.9790
Epoch 2/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0649 - accuracy: 0.9688
  32/1688 [..............................] - ETA: 2s - loss: 0.0654 - accuracy: 0.9766
  62/1688 [>.............................] - ETA: 2s - loss: 0.0622 - accuracy: 0.9788
  92/1688 [>.............................] - ETA: 2s - loss: 0.0587 - accuracy: 0.9806
 122/1688 [=>............................] - ETA: 2s - loss: 0.0609 - accuracy: 0.9787
 152/1688 [=>............................] - ETA: 2s - loss: 0.0651 - accuracy: 0.9782
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0618 - accuracy: 0.9796
 212/1688 [==>...........................] - ETA: 2s - loss: 0.0628 - accuracy: 0.9792
 242/1688 [===>..........................] - ETA: 2s - loss: 0.0668 - accuracy: 0.9784
 273/1688 [===>..........................] - ETA: 2s - loss: 0.0664 - accuracy: 0.9787
 303/1688 [====>.........................] - ETA: 2s - loss: 0.0685 - accuracy: 0.9783
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0695 - accuracy: 0.9782
 365/1688 [=====>........................] - ETA: 2s - loss: 0.0716 - accuracy: 0.9777
 395/1688 [======>.......................] - ETA: 2s - loss: 0.0717 - accuracy: 0.9771
 426/1688 [======>.......................] - ETA: 2s - loss: 0.0707 - accuracy: 0.9770
 456/1688 [=======>......................] - ETA: 2s - loss: 0.0703 - accuracy: 0.9770
 486/1688 [=======>......................] - ETA: 2s - loss: 0.0709 - accuracy: 0.9769
 516/1688 [========>.....................] - ETA: 1s - loss: 0.0713 - accuracy: 0.9767
 546/1688 [========>.....................] - ETA: 1s - loss: 0.0721 - accuracy: 0.9765
 576/1688 [=========>....................] - ETA: 1s - loss: 0.0732 - accuracy: 0.9762
 607/1688 [=========>....................] - ETA: 1s - loss: 0.0730 - accuracy: 0.9763
 638/1688 [==========>...................] - ETA: 1s - loss: 0.0729 - accuracy: 0.9763
 671/1688 [==========>...................] - ETA: 1s - loss: 0.0726 - accuracy: 0.9764
 702/1688 [===========>..................] - ETA: 1s - loss: 0.0735 - accuracy: 0.9761
 732/1688 [============>.................] - ETA: 1s - loss: 0.0750 - accuracy: 0.9757
 761/1688 [============>.................] - ETA: 1s - loss: 0.0751 - accuracy: 0.9756
 791/1688 [=============>................] - ETA: 1s - loss: 0.0752 - accuracy: 0.9757
 821/1688 [=============>................] - ETA: 1s - loss: 0.0755 - accuracy: 0.9757
 851/1688 [==============>...............] - ETA: 1s - loss: 0.0753 - accuracy: 0.9758
 881/1688 [==============>...............] - ETA: 1s - loss: 0.0748 - accuracy: 0.9759
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0749 - accuracy: 0.9759
 943/1688 [===============>..............] - ETA: 1s - loss: 0.0749 - accuracy: 0.9759
 973/1688 [================>.............] - ETA: 1s - loss: 0.0751 - accuracy: 0.9761
1004/1688 [================>.............] - ETA: 1s - loss: 0.0751 - accuracy: 0.9760
1034/1688 [=================>............] - ETA: 1s - loss: 0.0758 - accuracy: 0.9759
1064/1688 [=================>............] - ETA: 1s - loss: 0.0752 - accuracy: 0.9762
1095/1688 [==================>...........] - ETA: 0s - loss: 0.0756 - accuracy: 0.9760
1125/1688 [==================>...........] - ETA: 0s - loss: 0.0755 - accuracy: 0.9760
1155/1688 [===================>..........] - ETA: 0s - loss: 0.0749 - accuracy: 0.9761
1185/1688 [====================>.........] - ETA: 0s - loss: 0.0744 - accuracy: 0.9764
1215/1688 [====================>.........] - ETA: 0s - loss: 0.0743 - accuracy: 0.9765
1245/1688 [=====================>........] - ETA: 0s - loss: 0.0742 - accuracy: 0.9765
1275/1688 [=====================>........] - ETA: 0s - loss: 0.0735 - accuracy: 0.9767
1305/1688 [======================>.......] - ETA: 0s - loss: 0.0732 - accuracy: 0.9768
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0729 - accuracy: 0.9769
1366/1688 [=======================>......] - ETA: 0s - loss: 0.0726 - accuracy: 0.9769
1397/1688 [=======================>......] - ETA: 0s - loss: 0.0729 - accuracy: 0.9769
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0725 - accuracy: 0.9771
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0726 - accuracy: 0.9769
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0731 - accuracy: 0.9768
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0730 - accuracy: 0.9768
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0724 - accuracy: 0.9771
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0727 - accuracy: 0.9770
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0733 - accuracy: 0.9768
1638/1688 [============================>.] - ETA: 0s - loss: 0.0735 - accuracy: 0.9767
1668/1688 [============================>.] - ETA: 0s - loss: 0.0734 - accuracy: 0.9767
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0736 - accuracy: 0.9767 - val_loss: 0.0552 - val_accuracy: 0.9858
Epoch 3/10

   1/1688 [..............................] - ETA: 2s - loss: 0.1163 - accuracy: 0.9688
  34/1688 [..............................] - ETA: 2s - loss: 0.0780 - accuracy: 0.9770
  64/1688 [>.............................] - ETA: 2s - loss: 0.0629 - accuracy: 0.9819
  94/1688 [>.............................] - ETA: 2s - loss: 0.0534 - accuracy: 0.9840
 124/1688 [=>............................] - ETA: 2s - loss: 0.0508 - accuracy: 0.9841
 154/1688 [=>............................] - ETA: 2s - loss: 0.0501 - accuracy: 0.9842
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0488 - accuracy: 0.9840
 214/1688 [==>...........................] - ETA: 2s - loss: 0.0482 - accuracy: 0.9838
 244/1688 [===>..........................] - ETA: 2s - loss: 0.0460 - accuracy: 0.9845
 274/1688 [===>..........................] - ETA: 2s - loss: 0.0484 - accuracy: 0.9836
 304/1688 [====>.........................] - ETA: 2s - loss: 0.0492 - accuracy: 0.9834
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0472 - accuracy: 0.9840
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0483 - accuracy: 0.9839
 396/1688 [======>.......................] - ETA: 2s - loss: 0.0493 - accuracy: 0.9839
 428/1688 [======>.......................] - ETA: 2s - loss: 0.0483 - accuracy: 0.9841
 458/1688 [=======>......................] - ETA: 2s - loss: 0.0486 - accuracy: 0.9842
 489/1688 [=======>......................] - ETA: 2s - loss: 0.0495 - accuracy: 0.9841
 518/1688 [========>.....................] - ETA: 1s - loss: 0.0507 - accuracy: 0.9837
 549/1688 [========>.....................] - ETA: 1s - loss: 0.0500 - accuracy: 0.9838
 579/1688 [=========>....................] - ETA: 1s - loss: 0.0503 - accuracy: 0.9835
 610/1688 [=========>....................] - ETA: 1s - loss: 0.0498 - accuracy: 0.9838
 641/1688 [==========>...................] - ETA: 1s - loss: 0.0505 - accuracy: 0.9837
 671/1688 [==========>...................] - ETA: 1s - loss: 0.0514 - accuracy: 0.9836
 701/1688 [===========>..................] - ETA: 1s - loss: 0.0510 - accuracy: 0.9838
 731/1688 [===========>..................] - ETA: 1s - loss: 0.0510 - accuracy: 0.9838
 762/1688 [============>.................] - ETA: 1s - loss: 0.0509 - accuracy: 0.9838
 793/1688 [=============>................] - ETA: 1s - loss: 0.0518 - accuracy: 0.9837
 823/1688 [=============>................] - ETA: 1s - loss: 0.0524 - accuracy: 0.9835
 853/1688 [==============>...............] - ETA: 1s - loss: 0.0518 - accuracy: 0.9834
 883/1688 [==============>...............] - ETA: 1s - loss: 0.0526 - accuracy: 0.9833
 913/1688 [===============>..............] - ETA: 1s - loss: 0.0531 - accuracy: 0.9832
 943/1688 [===============>..............] - ETA: 1s - loss: 0.0529 - accuracy: 0.9833
 974/1688 [================>.............] - ETA: 1s - loss: 0.0532 - accuracy: 0.9833
1005/1688 [================>.............] - ETA: 1s - loss: 0.0526 - accuracy: 0.9836
1035/1688 [=================>............] - ETA: 1s - loss: 0.0524 - accuracy: 0.9837
1066/1688 [=================>............] - ETA: 1s - loss: 0.0527 - accuracy: 0.9837
1097/1688 [==================>...........] - ETA: 0s - loss: 0.0527 - accuracy: 0.9836
1126/1688 [===================>..........] - ETA: 0s - loss: 0.0536 - accuracy: 0.9833
1156/1688 [===================>..........] - ETA: 0s - loss: 0.0541 - accuracy: 0.9832
1186/1688 [====================>.........] - ETA: 0s - loss: 0.0541 - accuracy: 0.9832
1216/1688 [====================>.........] - ETA: 0s - loss: 0.0542 - accuracy: 0.9832
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0540 - accuracy: 0.9833
1277/1688 [=====================>........] - ETA: 0s - loss: 0.0538 - accuracy: 0.9833
1307/1688 [======================>.......] - ETA: 0s - loss: 0.0542 - accuracy: 0.9833
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0546 - accuracy: 0.9831
1368/1688 [=======================>......] - ETA: 0s - loss: 0.0542 - accuracy: 0.9832
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0549 - accuracy: 0.9830
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0555 - accuracy: 0.9829
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0558 - accuracy: 0.9828
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0559 - accuracy: 0.9828
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0563 - accuracy: 0.9827
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0560 - accuracy: 0.9828
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0561 - accuracy: 0.9827
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0560 - accuracy: 0.9827
1639/1688 [============================>.] - ETA: 0s - loss: 0.0555 - accuracy: 0.9829
1670/1688 [============================>.] - ETA: 0s - loss: 0.0554 - accuracy: 0.9830
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0557 - accuracy: 0.9829 - val_loss: 0.0695 - val_accuracy: 0.9810
Epoch 4/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0310 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0370 - accuracy: 0.9941
  62/1688 [>.............................] - ETA: 2s - loss: 0.0351 - accuracy: 0.9904
  91/1688 [>.............................] - ETA: 2s - loss: 0.0347 - accuracy: 0.9900
 122/1688 [=>............................] - ETA: 2s - loss: 0.0357 - accuracy: 0.9895
 152/1688 [=>............................] - ETA: 2s - loss: 0.0357 - accuracy: 0.9887
 183/1688 [==>...........................] - ETA: 2s - loss: 0.0361 - accuracy: 0.9891
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9881
 243/1688 [===>..........................] - ETA: 2s - loss: 0.0357 - accuracy: 0.9889
 274/1688 [===>..........................] - ETA: 2s - loss: 0.0350 - accuracy: 0.9888
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0366 - accuracy: 0.9883
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9878
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0373 - accuracy: 0.9880
 396/1688 [======>.......................] - ETA: 2s - loss: 0.0367 - accuracy: 0.9882
 426/1688 [======>.......................] - ETA: 2s - loss: 0.0380 - accuracy: 0.9880
 457/1688 [=======>......................] - ETA: 2s - loss: 0.0389 - accuracy: 0.9877
 488/1688 [=======>......................] - ETA: 2s - loss: 0.0388 - accuracy: 0.9878
 517/1688 [========>.....................] - ETA: 1s - loss: 0.0397 - accuracy: 0.9873
 548/1688 [========>.....................] - ETA: 1s - loss: 0.0393 - accuracy: 0.9874
 578/1688 [=========>....................] - ETA: 1s - loss: 0.0404 - accuracy: 0.9871
 608/1688 [=========>....................] - ETA: 1s - loss: 0.0405 - accuracy: 0.9870
 638/1688 [==========>...................] - ETA: 1s - loss: 0.0418 - accuracy: 0.9868
 668/1688 [==========>...................] - ETA: 1s - loss: 0.0421 - accuracy: 0.9868
 698/1688 [===========>..................] - ETA: 1s - loss: 0.0412 - accuracy: 0.9871
 728/1688 [===========>..................] - ETA: 1s - loss: 0.0419 - accuracy: 0.9869
 758/1688 [============>.................] - ETA: 1s - loss: 0.0421 - accuracy: 0.9868
 788/1688 [=============>................] - ETA: 1s - loss: 0.0428 - accuracy: 0.9866
 819/1688 [=============>................] - ETA: 1s - loss: 0.0430 - accuracy: 0.9865
 850/1688 [==============>...............] - ETA: 1s - loss: 0.0431 - accuracy: 0.9865
 880/1688 [==============>...............] - ETA: 1s - loss: 0.0427 - accuracy: 0.9865
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0426 - accuracy: 0.9865
 941/1688 [===============>..............] - ETA: 1s - loss: 0.0427 - accuracy: 0.9865
 971/1688 [================>.............] - ETA: 1s - loss: 0.0429 - accuracy: 0.9864
1001/1688 [================>.............] - ETA: 1s - loss: 0.0425 - accuracy: 0.9865
1031/1688 [=================>............] - ETA: 1s - loss: 0.0419 - accuracy: 0.9866
1061/1688 [=================>............] - ETA: 1s - loss: 0.0420 - accuracy: 0.9867
1091/1688 [==================>...........] - ETA: 1s - loss: 0.0432 - accuracy: 0.9863
1121/1688 [==================>...........] - ETA: 0s - loss: 0.0433 - accuracy: 0.9863
1151/1688 [===================>..........] - ETA: 0s - loss: 0.0434 - accuracy: 0.9864
1181/1688 [===================>..........] - ETA: 0s - loss: 0.0432 - accuracy: 0.9864
1212/1688 [====================>.........] - ETA: 0s - loss: 0.0439 - accuracy: 0.9863
1242/1688 [=====================>........] - ETA: 0s - loss: 0.0438 - accuracy: 0.9862
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0439 - accuracy: 0.9862
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0441 - accuracy: 0.9862
1333/1688 [======================>.......] - ETA: 0s - loss: 0.0440 - accuracy: 0.9862
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0440 - accuracy: 0.9863
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0443 - accuracy: 0.9861
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0440 - accuracy: 0.9862
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0440 - accuracy: 0.9862
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0440 - accuracy: 0.9862
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0436 - accuracy: 0.9863
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0434 - accuracy: 0.9864
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0434 - accuracy: 0.9863
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0431 - accuracy: 0.9864
1638/1688 [============================>.] - ETA: 0s - loss: 0.0431 - accuracy: 0.9864
1668/1688 [============================>.] - ETA: 0s - loss: 0.0430 - accuracy: 0.9864
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0430 - accuracy: 0.9864 - val_loss: 0.0560 - val_accuracy: 0.9845
Epoch 5/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0153 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0322 - accuracy: 0.9873
  62/1688 [>.............................] - ETA: 2s - loss: 0.0370 - accuracy: 0.9869
  92/1688 [>.............................] - ETA: 2s - loss: 0.0369 - accuracy: 0.9874
 122/1688 [=>............................] - ETA: 2s - loss: 0.0346 - accuracy: 0.9877
 153/1688 [=>............................] - ETA: 2s - loss: 0.0329 - accuracy: 0.9882
 183/1688 [==>...........................] - ETA: 2s - loss: 0.0337 - accuracy: 0.9877
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0319 - accuracy: 0.9883
 244/1688 [===>..........................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9889
 274/1688 [===>..........................] - ETA: 2s - loss: 0.0295 - accuracy: 0.9887
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0300 - accuracy: 0.9888
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0296 - accuracy: 0.9890
 365/1688 [=====>........................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9896
 395/1688 [======>.......................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9900
 426/1688 [======>.......................] - ETA: 2s - loss: 0.0294 - accuracy: 0.9893
 456/1688 [=======>......................] - ETA: 2s - loss: 0.0304 - accuracy: 0.9888
 486/1688 [=======>......................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9886
 515/1688 [========>.....................] - ETA: 1s - loss: 0.0307 - accuracy: 0.9887
 545/1688 [========>.....................] - ETA: 1s - loss: 0.0308 - accuracy: 0.9886
 575/1688 [=========>....................] - ETA: 1s - loss: 0.0309 - accuracy: 0.9886
 605/1688 [=========>....................] - ETA: 1s - loss: 0.0311 - accuracy: 0.9887
 636/1688 [==========>...................] - ETA: 1s - loss: 0.0309 - accuracy: 0.9887
 666/1688 [==========>...................] - ETA: 1s - loss: 0.0310 - accuracy: 0.9889
 696/1688 [===========>..................] - ETA: 1s - loss: 0.0306 - accuracy: 0.9890
 726/1688 [===========>..................] - ETA: 1s - loss: 0.0311 - accuracy: 0.9889
 756/1688 [============>.................] - ETA: 1s - loss: 0.0310 - accuracy: 0.9888
 786/1688 [============>.................] - ETA: 1s - loss: 0.0310 - accuracy: 0.9889
 817/1688 [=============>................] - ETA: 1s - loss: 0.0314 - accuracy: 0.9887
 847/1688 [==============>...............] - ETA: 1s - loss: 0.0316 - accuracy: 0.9887
 877/1688 [==============>...............] - ETA: 1s - loss: 0.0326 - accuracy: 0.9885
 908/1688 [===============>..............] - ETA: 1s - loss: 0.0328 - accuracy: 0.9882
 939/1688 [===============>..............] - ETA: 1s - loss: 0.0338 - accuracy: 0.9879
 968/1688 [================>.............] - ETA: 1s - loss: 0.0342 - accuracy: 0.9878
 999/1688 [================>.............] - ETA: 1s - loss: 0.0341 - accuracy: 0.9878
1030/1688 [=================>............] - ETA: 1s - loss: 0.0342 - accuracy: 0.9877
1060/1688 [=================>............] - ETA: 1s - loss: 0.0343 - accuracy: 0.9877
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0344 - accuracy: 0.9878
1119/1688 [==================>...........] - ETA: 0s - loss: 0.0347 - accuracy: 0.9877
1149/1688 [===================>..........] - ETA: 0s - loss: 0.0350 - accuracy: 0.9877
1180/1688 [===================>..........] - ETA: 0s - loss: 0.0348 - accuracy: 0.9878
1210/1688 [====================>.........] - ETA: 0s - loss: 0.0353 - accuracy: 0.9878
1240/1688 [=====================>........] - ETA: 0s - loss: 0.0357 - accuracy: 0.9879
1270/1688 [=====================>........] - ETA: 0s - loss: 0.0359 - accuracy: 0.9879
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0362 - accuracy: 0.9878
1331/1688 [======================>.......] - ETA: 0s - loss: 0.0360 - accuracy: 0.9880
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0357 - accuracy: 0.9881
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0354 - accuracy: 0.9882
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0355 - accuracy: 0.9883
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0356 - accuracy: 0.9882
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0354 - accuracy: 0.9883
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0355 - accuracy: 0.9883
1543/1688 [==========================>...] - ETA: 0s - loss: 0.0356 - accuracy: 0.9882
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0357 - accuracy: 0.9882
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0356 - accuracy: 0.9881
1635/1688 [============================>.] - ETA: 0s - loss: 0.0357 - accuracy: 0.9882
1665/1688 [============================>.] - ETA: 0s - loss: 0.0356 - accuracy: 0.9882
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0360 - accuracy: 0.9880 - val_loss: 0.0795 - val_accuracy: 0.9787
Epoch 6/10

   1/1688 [..............................] - ETA: 2s - loss: 0.1169 - accuracy: 0.9375
  32/1688 [..............................] - ETA: 2s - loss: 0.0304 - accuracy: 0.9902
  62/1688 [>.............................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9894
  92/1688 [>.............................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9915
 121/1688 [=>............................] - ETA: 2s - loss: 0.0252 - accuracy: 0.9920
 152/1688 [=>............................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9920
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0251 - accuracy: 0.9924
 212/1688 [==>...........................] - ETA: 2s - loss: 0.0256 - accuracy: 0.9922
 242/1688 [===>..........................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9925
 272/1688 [===>..........................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9928
 302/1688 [====>.........................] - ETA: 2s - loss: 0.0256 - accuracy: 0.9923
 333/1688 [====>.........................] - ETA: 2s - loss: 0.0253 - accuracy: 0.9921
 363/1688 [=====>........................] - ETA: 2s - loss: 0.0254 - accuracy: 0.9921
 392/1688 [=====>........................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9923
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0250 - accuracy: 0.9924
 454/1688 [=======>......................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9926
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0243 - accuracy: 0.9926
 517/1688 [========>.....................] - ETA: 1s - loss: 0.0240 - accuracy: 0.9925
 548/1688 [========>.....................] - ETA: 1s - loss: 0.0254 - accuracy: 0.9920
 579/1688 [=========>....................] - ETA: 1s - loss: 0.0250 - accuracy: 0.9921
 610/1688 [=========>....................] - ETA: 1s - loss: 0.0245 - accuracy: 0.9922
 640/1688 [==========>...................] - ETA: 1s - loss: 0.0254 - accuracy: 0.9918
 670/1688 [==========>...................] - ETA: 1s - loss: 0.0250 - accuracy: 0.9920
 701/1688 [===========>..................] - ETA: 1s - loss: 0.0252 - accuracy: 0.9919
 731/1688 [===========>..................] - ETA: 1s - loss: 0.0251 - accuracy: 0.9919
 761/1688 [============>.................] - ETA: 1s - loss: 0.0255 - accuracy: 0.9919
 791/1688 [=============>................] - ETA: 1s - loss: 0.0256 - accuracy: 0.9917
 821/1688 [=============>................] - ETA: 1s - loss: 0.0253 - accuracy: 0.9918
 851/1688 [==============>...............] - ETA: 1s - loss: 0.0255 - accuracy: 0.9917
 882/1688 [==============>...............] - ETA: 1s - loss: 0.0259 - accuracy: 0.9915
 912/1688 [===============>..............] - ETA: 1s - loss: 0.0259 - accuracy: 0.9916
 941/1688 [===============>..............] - ETA: 1s - loss: 0.0258 - accuracy: 0.9915
 971/1688 [================>.............] - ETA: 1s - loss: 0.0256 - accuracy: 0.9915
1001/1688 [================>.............] - ETA: 1s - loss: 0.0263 - accuracy: 0.9913
1031/1688 [=================>............] - ETA: 1s - loss: 0.0265 - accuracy: 0.9911
1061/1688 [=================>............] - ETA: 1s - loss: 0.0265 - accuracy: 0.9912
1091/1688 [==================>...........] - ETA: 1s - loss: 0.0272 - accuracy: 0.9911
1121/1688 [==================>...........] - ETA: 0s - loss: 0.0276 - accuracy: 0.9909
1151/1688 [===================>..........] - ETA: 0s - loss: 0.0280 - accuracy: 0.9907
1181/1688 [===================>..........] - ETA: 0s - loss: 0.0279 - accuracy: 0.9908
1212/1688 [====================>.........] - ETA: 0s - loss: 0.0282 - accuracy: 0.9906
1242/1688 [=====================>........] - ETA: 0s - loss: 0.0281 - accuracy: 0.9907
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0279 - accuracy: 0.9908
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0284 - accuracy: 0.9906
1333/1688 [======================>.......] - ETA: 0s - loss: 0.0284 - accuracy: 0.9906
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0287 - accuracy: 0.9905
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0290 - accuracy: 0.9904
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0292 - accuracy: 0.9904
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0292 - accuracy: 0.9904
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0293 - accuracy: 0.9903
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0294 - accuracy: 0.9902
1544/1688 [==========================>...] - ETA: 0s - loss: 0.0295 - accuracy: 0.9902
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0296 - accuracy: 0.9901
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0301 - accuracy: 0.9900
1635/1688 [============================>.] - ETA: 0s - loss: 0.0300 - accuracy: 0.9900
1665/1688 [============================>.] - ETA: 0s - loss: 0.0300 - accuracy: 0.9901
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0299 - accuracy: 0.9901 - val_loss: 0.0565 - val_accuracy: 0.9847
Epoch 7/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0020 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0292 - accuracy: 0.9854
  62/1688 [>.............................] - ETA: 2s - loss: 0.0330 - accuracy: 0.9854
  93/1688 [>.............................] - ETA: 2s - loss: 0.0312 - accuracy: 0.9876
 123/1688 [=>............................] - ETA: 2s - loss: 0.0314 - accuracy: 0.9878
 153/1688 [=>............................] - ETA: 2s - loss: 0.0285 - accuracy: 0.9892
 183/1688 [==>...........................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9896
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0280 - accuracy: 0.9896
 244/1688 [===>..........................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9899
 274/1688 [===>..........................] - ETA: 2s - loss: 0.0285 - accuracy: 0.9894
 304/1688 [====>.........................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9900
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9905
 365/1688 [=====>........................] - ETA: 2s - loss: 0.0259 - accuracy: 0.9908
 395/1688 [======>.......................] - ETA: 2s - loss: 0.0256 - accuracy: 0.9908
 426/1688 [======>.......................] - ETA: 2s - loss: 0.0255 - accuracy: 0.9909
 456/1688 [=======>......................] - ETA: 2s - loss: 0.0255 - accuracy: 0.9910
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0254 - accuracy: 0.9911
 516/1688 [========>.....................] - ETA: 1s - loss: 0.0253 - accuracy: 0.9911
 546/1688 [========>.....................] - ETA: 1s - loss: 0.0257 - accuracy: 0.9912
 578/1688 [=========>....................] - ETA: 1s - loss: 0.0256 - accuracy: 0.9912
 609/1688 [=========>....................] - ETA: 1s - loss: 0.0255 - accuracy: 0.9912
 640/1688 [==========>...................] - ETA: 1s - loss: 0.0258 - accuracy: 0.9910
 670/1688 [==========>...................] - ETA: 1s - loss: 0.0257 - accuracy: 0.9911
 700/1688 [===========>..................] - ETA: 1s - loss: 0.0252 - accuracy: 0.9912
 730/1688 [===========>..................] - ETA: 1s - loss: 0.0252 - accuracy: 0.9912
 761/1688 [============>.................] - ETA: 1s - loss: 0.0248 - accuracy: 0.9914
 790/1688 [=============>................] - ETA: 1s - loss: 0.0245 - accuracy: 0.9914
 820/1688 [=============>................] - ETA: 1s - loss: 0.0244 - accuracy: 0.9914
 850/1688 [==============>...............] - ETA: 1s - loss: 0.0248 - accuracy: 0.9913
 880/1688 [==============>...............] - ETA: 1s - loss: 0.0250 - accuracy: 0.9913
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0249 - accuracy: 0.9914
 942/1688 [===============>..............] - ETA: 1s - loss: 0.0253 - accuracy: 0.9912
 973/1688 [================>.............] - ETA: 1s - loss: 0.0248 - accuracy: 0.9914
1003/1688 [================>.............] - ETA: 1s - loss: 0.0253 - accuracy: 0.9913
1033/1688 [=================>............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9913
1063/1688 [=================>............] - ETA: 1s - loss: 0.0257 - accuracy: 0.9913
1093/1688 [==================>...........] - ETA: 0s - loss: 0.0260 - accuracy: 0.9913
1123/1688 [==================>...........] - ETA: 0s - loss: 0.0263 - accuracy: 0.9910
1154/1688 [===================>..........] - ETA: 0s - loss: 0.0265 - accuracy: 0.9910
1184/1688 [====================>.........] - ETA: 0s - loss: 0.0266 - accuracy: 0.9909
1214/1688 [====================>.........] - ETA: 0s - loss: 0.0264 - accuracy: 0.9910
1244/1688 [=====================>........] - ETA: 0s - loss: 0.0266 - accuracy: 0.9909
1275/1688 [=====================>........] - ETA: 0s - loss: 0.0266 - accuracy: 0.9910
1305/1688 [======================>.......] - ETA: 0s - loss: 0.0264 - accuracy: 0.9911
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0264 - accuracy: 0.9910
1366/1688 [=======================>......] - ETA: 0s - loss: 0.0265 - accuracy: 0.9910
1396/1688 [=======================>......] - ETA: 0s - loss: 0.0267 - accuracy: 0.9909
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0268 - accuracy: 0.9909
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0270 - accuracy: 0.9908
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0270 - accuracy: 0.9908
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0270 - accuracy: 0.9909
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0270 - accuracy: 0.9910
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0269 - accuracy: 0.9910
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0269 - accuracy: 0.9910
1640/1688 [============================>.] - ETA: 0s - loss: 0.0267 - accuracy: 0.9910
1671/1688 [============================>.] - ETA: 0s - loss: 0.0265 - accuracy: 0.9911
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0265 - accuracy: 0.9911 - val_loss: 0.0547 - val_accuracy: 0.9865
Epoch 8/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0347 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9971
  63/1688 [>.............................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9945
  93/1688 [>.............................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9940
 123/1688 [=>............................] - ETA: 2s - loss: 0.0168 - accuracy: 0.9947
 154/1688 [=>............................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9939
 185/1688 [==>...........................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9941
 215/1688 [==>...........................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9940
 245/1688 [===>..........................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9939
 276/1688 [===>..........................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9939
 306/1688 [====>.........................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9937
 337/1688 [====>.........................] - ETA: 2s - loss: 0.0197 - accuracy: 0.9933
 367/1688 [=====>........................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9934
 397/1688 [======>.......................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9930
 428/1688 [======>.......................] - ETA: 2s - loss: 0.0197 - accuracy: 0.9930
 458/1688 [=======>......................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9933
 488/1688 [=======>......................] - ETA: 1s - loss: 0.0188 - accuracy: 0.9933
 518/1688 [========>.....................] - ETA: 1s - loss: 0.0188 - accuracy: 0.9932
 548/1688 [========>.....................] - ETA: 1s - loss: 0.0202 - accuracy: 0.9928
 579/1688 [=========>....................] - ETA: 1s - loss: 0.0201 - accuracy: 0.9929
 609/1688 [=========>....................] - ETA: 1s - loss: 0.0199 - accuracy: 0.9930
 639/1688 [==========>...................] - ETA: 1s - loss: 0.0200 - accuracy: 0.9929
 669/1688 [==========>...................] - ETA: 1s - loss: 0.0198 - accuracy: 0.9930
 699/1688 [===========>..................] - ETA: 1s - loss: 0.0202 - accuracy: 0.9928
 729/1688 [===========>..................] - ETA: 1s - loss: 0.0203 - accuracy: 0.9928
 760/1688 [============>.................] - ETA: 1s - loss: 0.0204 - accuracy: 0.9926
 790/1688 [=============>................] - ETA: 1s - loss: 0.0203 - accuracy: 0.9927
 820/1688 [=============>................] - ETA: 1s - loss: 0.0204 - accuracy: 0.9927
 850/1688 [==============>...............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9927
 880/1688 [==============>...............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9927
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9927
 941/1688 [===============>..............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9927
 972/1688 [================>.............] - ETA: 1s - loss: 0.0203 - accuracy: 0.9928
1003/1688 [================>.............] - ETA: 1s - loss: 0.0203 - accuracy: 0.9928
1033/1688 [=================>............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9928
1063/1688 [=================>............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9927
1094/1688 [==================>...........] - ETA: 0s - loss: 0.0201 - accuracy: 0.9928
1125/1688 [==================>...........] - ETA: 0s - loss: 0.0205 - accuracy: 0.9927
1155/1688 [===================>..........] - ETA: 0s - loss: 0.0208 - accuracy: 0.9926
1185/1688 [====================>.........] - ETA: 0s - loss: 0.0208 - accuracy: 0.9925
1215/1688 [====================>.........] - ETA: 0s - loss: 0.0214 - accuracy: 0.9924
1245/1688 [=====================>........] - ETA: 0s - loss: 0.0215 - accuracy: 0.9924
1276/1688 [=====================>........] - ETA: 0s - loss: 0.0215 - accuracy: 0.9923
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0215 - accuracy: 0.9923
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0218 - accuracy: 0.9922
1368/1688 [=======================>......] - ETA: 0s - loss: 0.0218 - accuracy: 0.9922
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0220 - accuracy: 0.9921
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1459/1688 [========================>.....] - ETA: 0s - loss: 0.0219 - accuracy: 0.9920
1489/1688 [=========================>....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9920
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9920
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0217 - accuracy: 0.9921
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9920
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9920
1639/1688 [============================>.] - ETA: 0s - loss: 0.0222 - accuracy: 0.9919
1669/1688 [============================>.] - ETA: 0s - loss: 0.0220 - accuracy: 0.9920
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0221 - accuracy: 0.9920 - val_loss: 0.0554 - val_accuracy: 0.9873
Epoch 9/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0091 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9971
  62/1688 [>.............................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9960
  92/1688 [>.............................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9952
 122/1688 [=>............................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9949
 152/1688 [=>............................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9949
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9948
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9953
 243/1688 [===>..........................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9956
 274/1688 [===>..........................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9951
 304/1688 [====>.........................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9949
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9949
 364/1688 [=====>........................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9950
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9948
 425/1688 [======>.......................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9949
 455/1688 [=======>......................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9946
 485/1688 [=======>......................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9947
 516/1688 [========>.....................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9947
 546/1688 [========>.....................] - ETA: 1s - loss: 0.0150 - accuracy: 0.9942
 575/1688 [=========>....................] - ETA: 1s - loss: 0.0148 - accuracy: 0.9942
 605/1688 [=========>....................] - ETA: 1s - loss: 0.0147 - accuracy: 0.9943
 635/1688 [==========>...................] - ETA: 1s - loss: 0.0149 - accuracy: 0.9944
 665/1688 [==========>...................] - ETA: 1s - loss: 0.0151 - accuracy: 0.9944
 695/1688 [===========>..................] - ETA: 1s - loss: 0.0152 - accuracy: 0.9943
 725/1688 [===========>..................] - ETA: 1s - loss: 0.0152 - accuracy: 0.9942
 755/1688 [============>.................] - ETA: 1s - loss: 0.0153 - accuracy: 0.9941
 785/1688 [============>.................] - ETA: 1s - loss: 0.0158 - accuracy: 0.9939
 814/1688 [=============>................] - ETA: 1s - loss: 0.0157 - accuracy: 0.9940
 844/1688 [==============>...............] - ETA: 1s - loss: 0.0158 - accuracy: 0.9939
 874/1688 [==============>...............] - ETA: 1s - loss: 0.0163 - accuracy: 0.9937
 904/1688 [===============>..............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9936
 933/1688 [===============>..............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9937
 963/1688 [================>.............] - ETA: 1s - loss: 0.0168 - accuracy: 0.9936
 993/1688 [================>.............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9936
1023/1688 [=================>............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9935
1052/1688 [=================>............] - ETA: 1s - loss: 0.0183 - accuracy: 0.9932
1082/1688 [==================>...........] - ETA: 1s - loss: 0.0183 - accuracy: 0.9932
1111/1688 [==================>...........] - ETA: 0s - loss: 0.0183 - accuracy: 0.9932
1141/1688 [===================>..........] - ETA: 0s - loss: 0.0186 - accuracy: 0.9930
1170/1688 [===================>..........] - ETA: 0s - loss: 0.0189 - accuracy: 0.9929
1200/1688 [====================>.........] - ETA: 0s - loss: 0.0188 - accuracy: 0.9930
1230/1688 [====================>.........] - ETA: 0s - loss: 0.0189 - accuracy: 0.9930
1259/1688 [=====================>........] - ETA: 0s - loss: 0.0191 - accuracy: 0.9930
1289/1688 [=====================>........] - ETA: 0s - loss: 0.0190 - accuracy: 0.9930
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0193 - accuracy: 0.9929
1350/1688 [======================>.......] - ETA: 0s - loss: 0.0194 - accuracy: 0.9929
1379/1688 [=======================>......] - ETA: 0s - loss: 0.0194 - accuracy: 0.9929
1409/1688 [========================>.....] - ETA: 0s - loss: 0.0194 - accuracy: 0.9930
1438/1688 [========================>.....] - ETA: 0s - loss: 0.0196 - accuracy: 0.9929
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0198 - accuracy: 0.9928
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0199 - accuracy: 0.9928
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0200 - accuracy: 0.9928
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0202 - accuracy: 0.9927
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0203 - accuracy: 0.9927
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0204 - accuracy: 0.9927
1644/1688 [============================>.] - ETA: 0s - loss: 0.0205 - accuracy: 0.9927
1673/1688 [============================>.] - ETA: 0s - loss: 0.0207 - accuracy: 0.9926
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0206 - accuracy: 0.9926 - val_loss: 0.0606 - val_accuracy: 0.9865
Epoch 10/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0018 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9961
  62/1688 [>.............................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9965
  92/1688 [>.............................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9949
 123/1688 [=>............................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9952
 153/1688 [=>............................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9961
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0125 - accuracy: 0.9968
 215/1688 [==>...........................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9962
 245/1688 [===>..........................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9962
 275/1688 [===>..........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9960
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9960
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9959
 365/1688 [=====>........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9956
 395/1688 [======>.......................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9955
 425/1688 [======>.......................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9956
 455/1688 [=======>......................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9954
 485/1688 [=======>......................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9955
 515/1688 [========>.....................] - ETA: 1s - loss: 0.0148 - accuracy: 0.9954
 546/1688 [========>.....................] - ETA: 1s - loss: 0.0148 - accuracy: 0.9955
 576/1688 [=========>....................] - ETA: 1s - loss: 0.0149 - accuracy: 0.9953
 606/1688 [=========>....................] - ETA: 1s - loss: 0.0153 - accuracy: 0.9953
 636/1688 [==========>...................] - ETA: 1s - loss: 0.0150 - accuracy: 0.9953
 666/1688 [==========>...................] - ETA: 1s - loss: 0.0150 - accuracy: 0.9953
 696/1688 [===========>..................] - ETA: 1s - loss: 0.0152 - accuracy: 0.9951
 728/1688 [===========>..................] - ETA: 1s - loss: 0.0150 - accuracy: 0.9951
 758/1688 [============>.................] - ETA: 1s - loss: 0.0153 - accuracy: 0.9949
 788/1688 [=============>................] - ETA: 1s - loss: 0.0155 - accuracy: 0.9948
 819/1688 [=============>................] - ETA: 1s - loss: 0.0155 - accuracy: 0.9948
 850/1688 [==============>...............] - ETA: 1s - loss: 0.0152 - accuracy: 0.9950
 880/1688 [==============>...............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9950
 910/1688 [===============>..............] - ETA: 1s - loss: 0.0151 - accuracy: 0.9951
 940/1688 [===============>..............] - ETA: 1s - loss: 0.0156 - accuracy: 0.9948
 971/1688 [================>.............] - ETA: 1s - loss: 0.0159 - accuracy: 0.9948
1002/1688 [================>.............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9947
1032/1688 [=================>............] - ETA: 1s - loss: 0.0160 - accuracy: 0.9947
1063/1688 [=================>............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9946
1093/1688 [==================>...........] - ETA: 0s - loss: 0.0164 - accuracy: 0.9945
1124/1688 [==================>...........] - ETA: 0s - loss: 0.0163 - accuracy: 0.9946
1155/1688 [===================>..........] - ETA: 0s - loss: 0.0162 - accuracy: 0.9946
1185/1688 [====================>.........] - ETA: 0s - loss: 0.0162 - accuracy: 0.9945
1215/1688 [====================>.........] - ETA: 0s - loss: 0.0164 - accuracy: 0.9945
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0164 - accuracy: 0.9945
1277/1688 [=====================>........] - ETA: 0s - loss: 0.0163 - accuracy: 0.9944
1307/1688 [======================>.......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9944
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0167 - accuracy: 0.9943
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0169 - accuracy: 0.9942
1397/1688 [=======================>......] - ETA: 0s - loss: 0.0169 - accuracy: 0.9942
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0170 - accuracy: 0.9942
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0169 - accuracy: 0.9942
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0168 - accuracy: 0.9942
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0168 - accuracy: 0.9943
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0171 - accuracy: 0.9942
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0172 - accuracy: 0.9942
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0172 - accuracy: 0.9942
1640/1688 [============================>.] - ETA: 0s - loss: 0.0170 - accuracy: 0.9942
1670/1688 [============================>.] - ETA: 0s - loss: 0.0170 - accuracy: 0.9942
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0170 - accuracy: 0.9942 - val_loss: 0.0622 - val_accuracy: 0.9870

score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])

Test accuracy: 0.98580002784729

2. Quantize

2.1. 8-bit quantization

An Akida accelerator processes 8 or 4-bits integer activations and weights. Therefore, the floating point Keras model must be quantized in preparation to run on an Akida accelerator.

The QuantizeML quantize function can be used to quantize a Keras model for Akida. For this step in this example, an “8/8/8” quantization scheme will be applied to the floating point Keras model to produce 8-bit weights in the first layer, 8-bit weights in all other layers, and 8-bit activations.

The quantization process results in a Keras model with custom QuantizeML quantized layers substituted for the original Keras layers. All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function applies several transformations to the original model. For example, it folds the batch normalization layers into the corresponding neural layers. The new weights are computed according to this folding operation.

from quantizeml.models import quantize, QuantizationParams

qparams = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)
model_quantized = quantize(model_keras, qparams=qparams)
model_quantized.summary()

/usr/local/lib/python3.8/dist-packages/quantizeml/models/quantize.py:466: UserWarning: Quantizing per-axis with random calibration samples is not accurate.                       Set QuantizationParams.per_tensor_activations=True when calibrating with                        random samples.
  warnings.warn("Quantizing per-axis with random calibration samples is not accurate.\

   1/1024 [..............................] - ETA: 2:28
  58/1024 [>.............................] - ETA: 0s  
 116/1024 [==>...........................] - ETA: 0s
 172/1024 [====>.........................] - ETA: 0s
 229/1024 [=====>........................] - ETA: 0s
 287/1024 [=======>......................] - ETA: 0s
 345/1024 [=========>....................] - ETA: 0s
 404/1024 [==========>...................] - ETA: 0s
 463/1024 [============>.................] - ETA: 0s
 522/1024 [==============>...............] - ETA: 0s
 579/1024 [===============>..............] - ETA: 0s
 635/1024 [=================>............] - ETA: 0s
 693/1024 [===================>..........] - ETA: 0s
 751/1024 [=====================>........] - ETA: 0s
 808/1024 [======================>.......] - ETA: 0s
 866/1024 [========================>.....] - ETA: 0s
 924/1024 [==========================>...] - ETA: 0s
 982/1024 [===========================>..] - ETA: 0s
1024/1024 [==============================] - 1s 874us/step
Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling_input (InputLayer  [(None, 28, 28, 1)]      0
 )

 rescaling (QuantizedRescali  (None, 28, 28, 1)        0
 ng)

 conv2d (QuantizedConv2D)    (None, 13, 13, 32)        320

 re_lu (QuantizedReLU)       (None, 13, 13, 32)        64

 depthwise_conv2d (Quantized  (None, 7, 7, 32)         384
 DepthwiseConv2D)

 conv2d_1 (QuantizedConv2D)  (None, 7, 7, 64)          2112

 re_lu_1 (QuantizedReLU)     (None, 7, 7, 64)          128

 flatten (QuantizedFlatten)  (None, 3136)              0

 dense (QuantizedDense)      (None, 10)                31370

 dequantizer (Dequantizer)   (None, 10)                0

=================================================================
Total params: 34,378
Trainable params: 34,122
Non-trainable params: 256
_________________________________________________________________

Note

Note that the number of parameters for the floating and quantized models differs, a consequence of the BatchNormalization folding and the additional parameters added for quantization. For further details, please refer to their respective summary.

Check the quantized model accuracy.

def compile_evaluate(model):
    """ Compiles and evaluates the model, then return accuracy score. """
    model.compile(metrics=['accuracy'])
    return model.evaluate(x_test, y_test, verbose=0)[1]


print('Test accuracy after 8-bit quantization:', compile_evaluate(model_quantized))

Test accuracy after 8-bit quantization: 0.9810000061988831

2.2. Effect of calibration

The previous call to quantize was made with random samples for calibration (default parameters). While the observed drop in accuracy is minimal, that is around 1%, it can be worse on more complex models. Therefore, it is advised to use a set of real samples from the training set for calibration during a call to quantize. Note that this remains a calibration step rather than a training step in that no output labels are required. Furthermore, any relevant data could be used for calibration. The recommended settings for calibration that are widely used to obtain the zoo performance are:

• 1024 samples

• a batch size of 100

• 2 epochs

model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)

 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 978us/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 932us/step

Check the accuracy for the quantized and calibrated model.

print('Test accuracy after calibration:', compile_evaluate(model_quantized))

Test accuracy after calibration: 0.9851999878883362

Calibrating with real samples on this model recovers the initial float accuracy.

2.3. 4-bit quantization

The accuracy of the 8/8/8 quantized model is equal to that of the Keras floating point model. In some cases, a smaller memory size for the model is required. This can be accomplished through quantization of the model to smaller bitwidths.

The model will now be quantized to 8/4/4, that is 8-bit weights in the first layer with 4-bit weights and activations in all other layers. Such a quantization scheme will usually introduce a performance drop.

qparams = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)
model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)

 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 972us/step

Check the 4-bit quantized accuracy.

print('Test accuracy after 4-bit quantization:', compile_evaluate(model_quantized))

Test accuracy after 4-bit quantization: 0.9843000173568726

2.4. Model fine tuning (Quantization Aware Training)

When a model suffers from an accuracy drop after quantization, fine tuning or Quantization Aware Training (QAT) may recover some or all of the original performance.

Note that since this is a fine tuning step, both the number of epochs and learning rate are expected to be lower than during the initial float training.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-4),
    metrics=['accuracy'])

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)

Epoch 1/5

   1/1688 [..............................] - ETA: 1:05:48 - loss: 0.0087 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0159 - accuracy: 0.9952     
  25/1688 [..............................] - ETA: 7s - loss: 0.0160 - accuracy: 0.9937
  37/1688 [..............................] - ETA: 7s - loss: 0.0240 - accuracy: 0.9924
  49/1688 [..............................] - ETA: 7s - loss: 0.0238 - accuracy: 0.9917
  61/1688 [>.............................] - ETA: 7s - loss: 0.0224 - accuracy: 0.9923
  73/1688 [>.............................] - ETA: 7s - loss: 0.0203 - accuracy: 0.9927
  85/1688 [>.............................] - ETA: 7s - loss: 0.0184 - accuracy: 0.9934
  97/1688 [>.............................] - ETA: 7s - loss: 0.0165 - accuracy: 0.9939
 109/1688 [>.............................] - ETA: 7s - loss: 0.0162 - accuracy: 0.9943
 121/1688 [=>............................] - ETA: 6s - loss: 0.0161 - accuracy: 0.9946
 133/1688 [=>............................] - ETA: 6s - loss: 0.0151 - accuracy: 0.9951
 145/1688 [=>............................] - ETA: 6s - loss: 0.0146 - accuracy: 0.9953
 157/1688 [=>............................] - ETA: 6s - loss: 0.0146 - accuracy: 0.9952
 169/1688 [==>...........................] - ETA: 6s - loss: 0.0141 - accuracy: 0.9954
 181/1688 [==>...........................] - ETA: 6s - loss: 0.0137 - accuracy: 0.9955
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0142 - accuracy: 0.9955
 205/1688 [==>...........................] - ETA: 6s - loss: 0.0145 - accuracy: 0.9954
 217/1688 [==>...........................] - ETA: 6s - loss: 0.0144 - accuracy: 0.9952
 229/1688 [===>..........................] - ETA: 6s - loss: 0.0139 - accuracy: 0.9955
 241/1688 [===>..........................] - ETA: 6s - loss: 0.0135 - accuracy: 0.9956
 253/1688 [===>..........................] - ETA: 6s - loss: 0.0133 - accuracy: 0.9956
 265/1688 [===>..........................] - ETA: 6s - loss: 0.0133 - accuracy: 0.9955
 277/1688 [===>..........................] - ETA: 6s - loss: 0.0128 - accuracy: 0.9957
 289/1688 [====>.........................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9957
 301/1688 [====>.........................] - ETA: 6s - loss: 0.0129 - accuracy: 0.9956
 313/1688 [====>.........................] - ETA: 6s - loss: 0.0128 - accuracy: 0.9956
 325/1688 [====>.........................] - ETA: 6s - loss: 0.0128 - accuracy: 0.9955
 337/1688 [====>.........................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9954
 349/1688 [=====>........................] - ETA: 5s - loss: 0.0127 - accuracy: 0.9954
 361/1688 [=====>........................] - ETA: 5s - loss: 0.0127 - accuracy: 0.9955
 373/1688 [=====>........................] - ETA: 5s - loss: 0.0124 - accuracy: 0.9956
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9957
 397/1688 [======>.......................] - ETA: 5s - loss: 0.0120 - accuracy: 0.9958
 409/1688 [======>.......................] - ETA: 5s - loss: 0.0118 - accuracy: 0.9960
 421/1688 [======>.......................] - ETA: 5s - loss: 0.0116 - accuracy: 0.9961
 433/1688 [======>.......................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9962
 444/1688 [======>.......................] - ETA: 5s - loss: 0.0115 - accuracy: 0.9961
 456/1688 [=======>......................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9961
 468/1688 [=======>......................] - ETA: 5s - loss: 0.0115 - accuracy: 0.9961
 480/1688 [=======>......................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9962
 492/1688 [=======>......................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9963
 504/1688 [=======>......................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9963
 516/1688 [========>.....................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9964
 528/1688 [========>.....................] - ETA: 5s - loss: 0.0110 - accuracy: 0.9963
 540/1688 [========>.....................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9964
 552/1688 [========>.....................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9964
 564/1688 [=========>....................] - ETA: 4s - loss: 0.0111 - accuracy: 0.9963
 576/1688 [=========>....................] - ETA: 4s - loss: 0.0111 - accuracy: 0.9964
 588/1688 [=========>....................] - ETA: 4s - loss: 0.0109 - accuracy: 0.9965
 600/1688 [=========>....................] - ETA: 4s - loss: 0.0110 - accuracy: 0.9965
 612/1688 [=========>....................] - ETA: 4s - loss: 0.0111 - accuracy: 0.9964
 624/1688 [==========>...................] - ETA: 4s - loss: 0.0110 - accuracy: 0.9964
 636/1688 [==========>...................] - ETA: 4s - loss: 0.0110 - accuracy: 0.9964
 648/1688 [==========>...................] - ETA: 4s - loss: 0.0110 - accuracy: 0.9964
 660/1688 [==========>...................] - ETA: 4s - loss: 0.0109 - accuracy: 0.9964
 672/1688 [==========>...................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9965
 684/1688 [===========>..................] - ETA: 4s - loss: 0.0108 - accuracy: 0.9964
 696/1688 [===========>..................] - ETA: 4s - loss: 0.0108 - accuracy: 0.9964
 708/1688 [===========>..................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9964
 720/1688 [===========>..................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9965
 732/1688 [============>.................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9965
 744/1688 [============>.................] - ETA: 4s - loss: 0.0108 - accuracy: 0.9964
 756/1688 [============>.................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9964
 767/1688 [============>.................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9965
 779/1688 [============>.................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9965
 791/1688 [=============>................] - ETA: 3s - loss: 0.0105 - accuracy: 0.9965
 803/1688 [=============>................] - ETA: 3s - loss: 0.0104 - accuracy: 0.9965
 815/1688 [=============>................] - ETA: 3s - loss: 0.0104 - accuracy: 0.9966
 827/1688 [=============>................] - ETA: 3s - loss: 0.0103 - accuracy: 0.9966
 839/1688 [=============>................] - ETA: 3s - loss: 0.0104 - accuracy: 0.9966
 851/1688 [==============>...............] - ETA: 3s - loss: 0.0103 - accuracy: 0.9966
 863/1688 [==============>...............] - ETA: 3s - loss: 0.0104 - accuracy: 0.9966
 875/1688 [==============>...............] - ETA: 3s - loss: 0.0104 - accuracy: 0.9966
 887/1688 [==============>...............] - ETA: 3s - loss: 0.0103 - accuracy: 0.9967
 899/1688 [==============>...............] - ETA: 3s - loss: 0.0103 - accuracy: 0.9967
 911/1688 [===============>..............] - ETA: 3s - loss: 0.0103 - accuracy: 0.9967
 923/1688 [===============>..............] - ETA: 3s - loss: 0.0103 - accuracy: 0.9967
 935/1688 [===============>..............] - ETA: 3s - loss: 0.0102 - accuracy: 0.9967
 947/1688 [===============>..............] - ETA: 3s - loss: 0.0102 - accuracy: 0.9967
 959/1688 [================>.............] - ETA: 3s - loss: 0.0101 - accuracy: 0.9967
 971/1688 [================>.............] - ETA: 3s - loss: 0.0102 - accuracy: 0.9967
 983/1688 [================>.............] - ETA: 3s - loss: 0.0103 - accuracy: 0.9967
 995/1688 [================>.............] - ETA: 3s - loss: 0.0103 - accuracy: 0.9966
1007/1688 [================>.............] - ETA: 3s - loss: 0.0102 - accuracy: 0.9966
1019/1688 [=================>............] - ETA: 2s - loss: 0.0102 - accuracy: 0.9967
1031/1688 [=================>............] - ETA: 2s - loss: 0.0102 - accuracy: 0.9966
1043/1688 [=================>............] - ETA: 2s - loss: 0.0102 - accuracy: 0.9967
1055/1688 [=================>............] - ETA: 2s - loss: 0.0101 - accuracy: 0.9967
1067/1688 [=================>............] - ETA: 2s - loss: 0.0103 - accuracy: 0.9966
1079/1688 [==================>...........] - ETA: 2s - loss: 0.0103 - accuracy: 0.9966
1091/1688 [==================>...........] - ETA: 2s - loss: 0.0103 - accuracy: 0.9966
1103/1688 [==================>...........] - ETA: 2s - loss: 0.0103 - accuracy: 0.9967
1115/1688 [==================>...........] - ETA: 2s - loss: 0.0103 - accuracy: 0.9967
1127/1688 [===================>..........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9967
1139/1688 [===================>..........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9967
1151/1688 [===================>..........] - ETA: 2s - loss: 0.0101 - accuracy: 0.9968
1163/1688 [===================>..........] - ETA: 2s - loss: 0.0101 - accuracy: 0.9968
1175/1688 [===================>..........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9968
1187/1688 [====================>.........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9968
1199/1688 [====================>.........] - ETA: 2s - loss: 0.0101 - accuracy: 0.9968
1211/1688 [====================>.........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9968
1223/1688 [====================>.........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9968
1235/1688 [====================>.........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9968
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0100 - accuracy: 0.9968
1259/1688 [=====================>........] - ETA: 1s - loss: 0.0099 - accuracy: 0.9968
1271/1688 [=====================>........] - ETA: 1s - loss: 0.0100 - accuracy: 0.9968
1283/1688 [=====================>........] - ETA: 1s - loss: 0.0100 - accuracy: 0.9968
1295/1688 [======================>.......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9968
1307/1688 [======================>.......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9967
1319/1688 [======================>.......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9968
1331/1688 [======================>.......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9968
1343/1688 [======================>.......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9968
1355/1688 [=======================>......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9968
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9968
1379/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9969
1391/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9969
1403/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9968
1415/1688 [========================>.....] - ETA: 1s - loss: 0.0101 - accuracy: 0.9968
1427/1688 [========================>.....] - ETA: 1s - loss: 0.0101 - accuracy: 0.9968
1439/1688 [========================>.....] - ETA: 1s - loss: 0.0101 - accuracy: 0.9968
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0101 - accuracy: 0.9969
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0100 - accuracy: 0.9969
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0101 - accuracy: 0.9968
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0100 - accuracy: 0.9969
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0100 - accuracy: 0.9969
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0100 - accuracy: 0.9969
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0099 - accuracy: 0.9969
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0099 - accuracy: 0.9969
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0101 - accuracy: 0.9968
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0100 - accuracy: 0.9969
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0100 - accuracy: 0.9969
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0100 - accuracy: 0.9969
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0099 - accuracy: 0.9969
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0099 - accuracy: 0.9969
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0098 - accuracy: 0.9970
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0098 - accuracy: 0.9970
1643/1688 [============================>.] - ETA: 0s - loss: 0.0099 - accuracy: 0.9969
1655/1688 [============================>.] - ETA: 0s - loss: 0.0099 - accuracy: 0.9969
1667/1688 [============================>.] - ETA: 0s - loss: 0.0099 - accuracy: 0.9969
1679/1688 [============================>.] - ETA: 0s - loss: 0.0098 - accuracy: 0.9969
1688/1688 [==============================] - 11s 5ms/step - loss: 0.0098 - accuracy: 0.9969 - val_loss: 0.0573 - val_accuracy: 0.9875
Epoch 2/5

   1/1688 [..............................] - ETA: 7s - loss: 7.0202e-05 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9976    
  25/1688 [..............................] - ETA: 7s - loss: 0.0072 - accuracy: 0.9975
  37/1688 [..............................] - ETA: 7s - loss: 0.0083 - accuracy: 0.9958
  49/1688 [..............................] - ETA: 7s - loss: 0.0072 - accuracy: 0.9968
  61/1688 [>.............................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9974
  73/1688 [>.............................] - ETA: 7s - loss: 0.0058 - accuracy: 0.9979
  85/1688 [>.............................] - ETA: 7s - loss: 0.0063 - accuracy: 0.9974
  97/1688 [>.............................] - ETA: 7s - loss: 0.0065 - accuracy: 0.9974
 109/1688 [>.............................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9977
 121/1688 [=>............................] - ETA: 6s - loss: 0.0071 - accuracy: 0.9977
 133/1688 [=>............................] - ETA: 6s - loss: 0.0075 - accuracy: 0.9977
 145/1688 [=>............................] - ETA: 6s - loss: 0.0075 - accuracy: 0.9976
 157/1688 [=>............................] - ETA: 6s - loss: 0.0077 - accuracy: 0.9974
 169/1688 [==>...........................] - ETA: 6s - loss: 0.0074 - accuracy: 0.9976
 181/1688 [==>...........................] - ETA: 6s - loss: 0.0072 - accuracy: 0.9978
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0072 - accuracy: 0.9977
 205/1688 [==>...........................] - ETA: 6s - loss: 0.0073 - accuracy: 0.9977
 217/1688 [==>...........................] - ETA: 6s - loss: 0.0073 - accuracy: 0.9977
 229/1688 [===>..........................] - ETA: 6s - loss: 0.0079 - accuracy: 0.9975
 241/1688 [===>..........................] - ETA: 6s - loss: 0.0080 - accuracy: 0.9975
 253/1688 [===>..........................] - ETA: 6s - loss: 0.0080 - accuracy: 0.9977
 265/1688 [===>..........................] - ETA: 6s - loss: 0.0077 - accuracy: 0.9978
 277/1688 [===>..........................] - ETA: 6s - loss: 0.0078 - accuracy: 0.9977
 289/1688 [====>.........................] - ETA: 6s - loss: 0.0077 - accuracy: 0.9977
 301/1688 [====>.........................] - ETA: 6s - loss: 0.0078 - accuracy: 0.9977
 313/1688 [====>.........................] - ETA: 6s - loss: 0.0077 - accuracy: 0.9978
 325/1688 [====>.........................] - ETA: 6s - loss: 0.0076 - accuracy: 0.9978
 337/1688 [====>.........................] - ETA: 5s - loss: 0.0077 - accuracy: 0.9978
 349/1688 [=====>........................] - ETA: 5s - loss: 0.0077 - accuracy: 0.9978
 361/1688 [=====>........................] - ETA: 5s - loss: 0.0077 - accuracy: 0.9978
 373/1688 [=====>........................] - ETA: 5s - loss: 0.0075 - accuracy: 0.9979
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0075 - accuracy: 0.9979
 397/1688 [======>.......................] - ETA: 5s - loss: 0.0075 - accuracy: 0.9978
 409/1688 [======>.......................] - ETA: 5s - loss: 0.0074 - accuracy: 0.9979
 421/1688 [======>.......................] - ETA: 5s - loss: 0.0075 - accuracy: 0.9978
 433/1688 [======>.......................] - ETA: 5s - loss: 0.0075 - accuracy: 0.9978
 445/1688 [======>.......................] - ETA: 5s - loss: 0.0073 - accuracy: 0.9978
 457/1688 [=======>......................] - ETA: 5s - loss: 0.0073 - accuracy: 0.9978
 469/1688 [=======>......................] - ETA: 5s - loss: 0.0073 - accuracy: 0.9978
 481/1688 [=======>......................] - ETA: 5s - loss: 0.0074 - accuracy: 0.9977
 493/1688 [=======>......................] - ETA: 5s - loss: 0.0075 - accuracy: 0.9977
 505/1688 [=======>......................] - ETA: 5s - loss: 0.0074 - accuracy: 0.9977
 517/1688 [========>.....................] - ETA: 5s - loss: 0.0074 - accuracy: 0.9978
 529/1688 [========>.....................] - ETA: 5s - loss: 0.0074 - accuracy: 0.9978
 541/1688 [========>.....................] - ETA: 5s - loss: 0.0074 - accuracy: 0.9977
 553/1688 [========>.....................] - ETA: 5s - loss: 0.0077 - accuracy: 0.9976
 565/1688 [=========>....................] - ETA: 4s - loss: 0.0075 - accuracy: 0.9977
 577/1688 [=========>....................] - ETA: 4s - loss: 0.0076 - accuracy: 0.9977
 589/1688 [=========>....................] - ETA: 4s - loss: 0.0075 - accuracy: 0.9978
 601/1688 [=========>....................] - ETA: 4s - loss: 0.0075 - accuracy: 0.9978
 613/1688 [=========>....................] - ETA: 4s - loss: 0.0075 - accuracy: 0.9977
 625/1688 [==========>...................] - ETA: 4s - loss: 0.0077 - accuracy: 0.9976
 637/1688 [==========>...................] - ETA: 4s - loss: 0.0078 - accuracy: 0.9976
 649/1688 [==========>...................] - ETA: 4s - loss: 0.0079 - accuracy: 0.9975
 661/1688 [==========>...................] - ETA: 4s - loss: 0.0079 - accuracy: 0.9975
 673/1688 [==========>...................] - ETA: 4s - loss: 0.0078 - accuracy: 0.9976
 685/1688 [===========>..................] - ETA: 4s - loss: 0.0077 - accuracy: 0.9976
 697/1688 [===========>..................] - ETA: 4s - loss: 0.0077 - accuracy: 0.9977
 709/1688 [===========>..................] - ETA: 4s - loss: 0.0076 - accuracy: 0.9977
 721/1688 [===========>..................] - ETA: 4s - loss: 0.0075 - accuracy: 0.9977
 733/1688 [============>.................] - ETA: 4s - loss: 0.0075 - accuracy: 0.9978
 745/1688 [============>.................] - ETA: 4s - loss: 0.0075 - accuracy: 0.9978
 757/1688 [============>.................] - ETA: 4s - loss: 0.0074 - accuracy: 0.9979
 769/1688 [============>.................] - ETA: 4s - loss: 0.0074 - accuracy: 0.9978
 781/1688 [============>.................] - ETA: 4s - loss: 0.0073 - accuracy: 0.9979
 793/1688 [=============>................] - ETA: 3s - loss: 0.0075 - accuracy: 0.9979
 805/1688 [=============>................] - ETA: 3s - loss: 0.0075 - accuracy: 0.9979
 817/1688 [=============>................] - ETA: 3s - loss: 0.0075 - accuracy: 0.9979
 829/1688 [=============>................] - ETA: 3s - loss: 0.0075 - accuracy: 0.9979
 841/1688 [=============>................] - ETA: 3s - loss: 0.0075 - accuracy: 0.9979
 853/1688 [==============>...............] - ETA: 3s - loss: 0.0075 - accuracy: 0.9979
 865/1688 [==============>...............] - ETA: 3s - loss: 0.0074 - accuracy: 0.9979
 877/1688 [==============>...............] - ETA: 3s - loss: 0.0074 - accuracy: 0.9980
 889/1688 [==============>...............] - ETA: 3s - loss: 0.0074 - accuracy: 0.9980
 901/1688 [===============>..............] - ETA: 3s - loss: 0.0073 - accuracy: 0.9980
 913/1688 [===============>..............] - ETA: 3s - loss: 0.0073 - accuracy: 0.9980
 925/1688 [===============>..............] - ETA: 3s - loss: 0.0072 - accuracy: 0.9980
 937/1688 [===============>..............] - ETA: 3s - loss: 0.0072 - accuracy: 0.9980
 949/1688 [===============>..............] - ETA: 3s - loss: 0.0072 - accuracy: 0.9981
 961/1688 [================>.............] - ETA: 3s - loss: 0.0071 - accuracy: 0.9981
 973/1688 [================>.............] - ETA: 3s - loss: 0.0071 - accuracy: 0.9981
 985/1688 [================>.............] - ETA: 3s - loss: 0.0071 - accuracy: 0.9981
 997/1688 [================>.............] - ETA: 3s - loss: 0.0071 - accuracy: 0.9981
1009/1688 [================>.............] - ETA: 3s - loss: 0.0071 - accuracy: 0.9980
1021/1688 [=================>............] - ETA: 2s - loss: 0.0070 - accuracy: 0.9981
1033/1688 [=================>............] - ETA: 2s - loss: 0.0070 - accuracy: 0.9981
1045/1688 [=================>............] - ETA: 2s - loss: 0.0070 - accuracy: 0.9981
1057/1688 [=================>............] - ETA: 2s - loss: 0.0070 - accuracy: 0.9981
1069/1688 [=================>............] - ETA: 2s - loss: 0.0069 - accuracy: 0.9982
1081/1688 [==================>...........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9981
1093/1688 [==================>...........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9981
1105/1688 [==================>...........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9981
1117/1688 [==================>...........] - ETA: 2s - loss: 0.0068 - accuracy: 0.9982
1129/1688 [===================>..........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9981
1141/1688 [===================>..........] - ETA: 2s - loss: 0.0068 - accuracy: 0.9982
1153/1688 [===================>..........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9981
1165/1688 [===================>..........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9981
1177/1688 [===================>..........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9982
1189/1688 [====================>.........] - ETA: 2s - loss: 0.0068 - accuracy: 0.9982
1201/1688 [====================>.........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9982
1213/1688 [====================>.........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9981
1225/1688 [====================>.........] - ETA: 2s - loss: 0.0069 - accuracy: 0.9982
1237/1688 [====================>.........] - ETA: 1s - loss: 0.0068 - accuracy: 0.9982
1249/1688 [=====================>........] - ETA: 1s - loss: 0.0068 - accuracy: 0.9982
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1273/1688 [=====================>........] - ETA: 1s - loss: 0.0069 - accuracy: 0.9982
1285/1688 [=====================>........] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1297/1688 [======================>.......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1309/1688 [======================>.......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1321/1688 [======================>.......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1333/1688 [======================>.......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1345/1688 [======================>.......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1357/1688 [=======================>......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1369/1688 [=======================>......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1381/1688 [=======================>......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1393/1688 [=======================>......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1405/1688 [=======================>......] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1417/1688 [========================>.....] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1429/1688 [========================>.....] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1441/1688 [========================>.....] - ETA: 1s - loss: 0.0070 - accuracy: 0.9981
1453/1688 [========================>.....] - ETA: 1s - loss: 0.0069 - accuracy: 0.9981
1465/1688 [=========================>....] - ETA: 0s - loss: 0.0070 - accuracy: 0.9981
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0069 - accuracy: 0.9981
1489/1688 [=========================>....] - ETA: 0s - loss: 0.0069 - accuracy: 0.9981
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0069 - accuracy: 0.9981
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0068 - accuracy: 0.9982
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0068 - accuracy: 0.9982
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0069 - accuracy: 0.9982
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0069 - accuracy: 0.9981
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0069 - accuracy: 0.9981
1573/1688 [==========================>...] - ETA: 0s - loss: 0.0069 - accuracy: 0.9982
1585/1688 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9982
1597/1688 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9982
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0068 - accuracy: 0.9982
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0068 - accuracy: 0.9982
1633/1688 [============================>.] - ETA: 0s - loss: 0.0068 - accuracy: 0.9982
1645/1688 [============================>.] - ETA: 0s - loss: 0.0068 - accuracy: 0.9982
1657/1688 [============================>.] - ETA: 0s - loss: 0.0068 - accuracy: 0.9982
1669/1688 [============================>.] - ETA: 0s - loss: 0.0068 - accuracy: 0.9982
1681/1688 [============================>.] - ETA: 0s - loss: 0.0067 - accuracy: 0.9982
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0067 - accuracy: 0.9982 - val_loss: 0.0551 - val_accuracy: 0.9882
Epoch 3/5

   1/1688 [..............................] - ETA: 7s - loss: 1.6446e-04 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9976    
  25/1688 [..............................] - ETA: 7s - loss: 0.0070 - accuracy: 0.9975
  37/1688 [..............................] - ETA: 7s - loss: 0.0075 - accuracy: 0.9975
  49/1688 [..............................] - ETA: 7s - loss: 0.0073 - accuracy: 0.9981
  61/1688 [>.............................] - ETA: 7s - loss: 0.0069 - accuracy: 0.9985
  73/1688 [>.............................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9987
  85/1688 [>.............................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9985
  97/1688 [>.............................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9981
 109/1688 [>.............................] - ETA: 6s - loss: 0.0069 - accuracy: 0.9980
 121/1688 [=>............................] - ETA: 6s - loss: 0.0069 - accuracy: 0.9979
 133/1688 [=>............................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9981
 145/1688 [=>............................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9978
 157/1688 [=>............................] - ETA: 6s - loss: 0.0066 - accuracy: 0.9980
 169/1688 [==>...........................] - ETA: 6s - loss: 0.0070 - accuracy: 0.9978
 181/1688 [==>...........................] - ETA: 6s - loss: 0.0071 - accuracy: 0.9978
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0071 - accuracy: 0.9979
 205/1688 [==>...........................] - ETA: 6s - loss: 0.0069 - accuracy: 0.9980
 217/1688 [==>...........................] - ETA: 6s - loss: 0.0070 - accuracy: 0.9980
 229/1688 [===>..........................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9981
 241/1688 [===>..........................] - ETA: 6s - loss: 0.0067 - accuracy: 0.9981
 253/1688 [===>..........................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9981
 265/1688 [===>..........................] - ETA: 6s - loss: 0.0063 - accuracy: 0.9982
 277/1688 [===>..........................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9982
 289/1688 [====>.........................] - ETA: 6s - loss: 0.0063 - accuracy: 0.9983
 301/1688 [====>.........................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9983
 313/1688 [====>.........................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9984
 325/1688 [====>.........................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9985
 337/1688 [====>.........................] - ETA: 5s - loss: 0.0059 - accuracy: 0.9985
 349/1688 [=====>........................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9986
 361/1688 [=====>........................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9986
 373/1688 [=====>........................] - ETA: 5s - loss: 0.0059 - accuracy: 0.9986
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9986
 397/1688 [======>.......................] - ETA: 5s - loss: 0.0059 - accuracy: 0.9985
 409/1688 [======>.......................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9985
 421/1688 [======>.......................] - ETA: 5s - loss: 0.0059 - accuracy: 0.9984
 433/1688 [======>.......................] - ETA: 5s - loss: 0.0059 - accuracy: 0.9985
 445/1688 [======>.......................] - ETA: 5s - loss: 0.0059 - accuracy: 0.9985
 457/1688 [=======>......................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9986
 469/1688 [=======>......................] - ETA: 5s - loss: 0.0057 - accuracy: 0.9986
 481/1688 [=======>......................] - ETA: 5s - loss: 0.0057 - accuracy: 0.9986
 493/1688 [=======>......................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9986
 505/1688 [=======>......................] - ETA: 5s - loss: 0.0057 - accuracy: 0.9986
 517/1688 [========>.....................] - ETA: 5s - loss: 0.0057 - accuracy: 0.9987
 529/1688 [========>.....................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9986
 541/1688 [========>.....................] - ETA: 5s - loss: 0.0057 - accuracy: 0.9986
 553/1688 [========>.....................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9985
 565/1688 [=========>....................] - ETA: 4s - loss: 0.0060 - accuracy: 0.9985
 577/1688 [=========>....................] - ETA: 4s - loss: 0.0060 - accuracy: 0.9984
 589/1688 [=========>....................] - ETA: 4s - loss: 0.0061 - accuracy: 0.9984
 601/1688 [=========>....................] - ETA: 4s - loss: 0.0062 - accuracy: 0.9984
 613/1688 [=========>....................] - ETA: 4s - loss: 0.0062 - accuracy: 0.9984
 625/1688 [==========>...................] - ETA: 4s - loss: 0.0061 - accuracy: 0.9984
 637/1688 [==========>...................] - ETA: 4s - loss: 0.0061 - accuracy: 0.9984
 649/1688 [==========>...................] - ETA: 4s - loss: 0.0060 - accuracy: 0.9985
 661/1688 [==========>...................] - ETA: 4s - loss: 0.0060 - accuracy: 0.9985
 673/1688 [==========>...................] - ETA: 4s - loss: 0.0060 - accuracy: 0.9984
 685/1688 [===========>..................] - ETA: 4s - loss: 0.0060 - accuracy: 0.9984
 697/1688 [===========>..................] - ETA: 4s - loss: 0.0060 - accuracy: 0.9985
 709/1688 [===========>..................] - ETA: 4s - loss: 0.0060 - accuracy: 0.9985
 721/1688 [===========>..................] - ETA: 4s - loss: 0.0059 - accuracy: 0.9985
 733/1688 [============>.................] - ETA: 4s - loss: 0.0059 - accuracy: 0.9986
 745/1688 [============>.................] - ETA: 4s - loss: 0.0058 - accuracy: 0.9986
 757/1688 [============>.................] - ETA: 4s - loss: 0.0058 - accuracy: 0.9986
 769/1688 [============>.................] - ETA: 4s - loss: 0.0058 - accuracy: 0.9986
 781/1688 [============>.................] - ETA: 4s - loss: 0.0057 - accuracy: 0.9986
 793/1688 [=============>................] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 805/1688 [=============>................] - ETA: 3s - loss: 0.0058 - accuracy: 0.9986
 817/1688 [=============>................] - ETA: 3s - loss: 0.0057 - accuracy: 0.9986
 829/1688 [=============>................] - ETA: 3s - loss: 0.0058 - accuracy: 0.9986
 841/1688 [=============>................] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 853/1688 [==============>...............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 865/1688 [==============>...............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 877/1688 [==============>...............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 889/1688 [==============>...............] - ETA: 3s - loss: 0.0058 - accuracy: 0.9987
 901/1688 [===============>..............] - ETA: 3s - loss: 0.0058 - accuracy: 0.9987
 913/1688 [===============>..............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 925/1688 [===============>..............] - ETA: 3s - loss: 0.0058 - accuracy: 0.9986
 937/1688 [===============>..............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 949/1688 [===============>..............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 961/1688 [================>.............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 973/1688 [================>.............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 985/1688 [================>.............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
 997/1688 [================>.............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9987
1009/1688 [================>.............] - ETA: 3s - loss: 0.0056 - accuracy: 0.9987
1021/1688 [=================>............] - ETA: 2s - loss: 0.0057 - accuracy: 0.9987
1033/1688 [=================>............] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1045/1688 [=================>............] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1057/1688 [=================>............] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1069/1688 [=================>............] - ETA: 2s - loss: 0.0059 - accuracy: 0.9985
1081/1688 [==================>...........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9986
1093/1688 [==================>...........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9986
1105/1688 [==================>...........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1117/1688 [==================>...........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1129/1688 [===================>..........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1141/1688 [===================>..........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9986
1153/1688 [===================>..........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1165/1688 [===================>..........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1177/1688 [===================>..........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1189/1688 [====================>.........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1201/1688 [====================>.........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9986
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9986
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9986
1236/1688 [====================>.........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9986
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0058 - accuracy: 0.9986
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0058 - accuracy: 0.9987
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0058 - accuracy: 0.9986
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0058 - accuracy: 0.9987
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1308/1688 [======================>.......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1320/1688 [======================>.......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1380/1688 [=======================>......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1404/1688 [=======================>......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1416/1688 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1440/1688 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9986
1452/1688 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1464/1688 [=========================>....] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0056 - accuracy: 0.9987
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0056 - accuracy: 0.9987
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0056 - accuracy: 0.9987
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0056 - accuracy: 0.9987
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0056 - accuracy: 0.9987
1632/1688 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1644/1688 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1656/1688 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1668/1688 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1680/1688 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9987
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0057 - accuracy: 0.9987 - val_loss: 0.0546 - val_accuracy: 0.9885
Epoch 4/5

   1/1688 [..............................] - ETA: 7s - loss: 4.5082e-04 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0034 - accuracy: 1.0000    
  25/1688 [..............................] - ETA: 7s - loss: 0.0027 - accuracy: 1.0000
  37/1688 [..............................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9992
  49/1688 [..............................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9994
  61/1688 [>.............................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9995
  73/1688 [>.............................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9996
  85/1688 [>.............................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9993
  97/1688 [>.............................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9994
 109/1688 [>.............................] - ETA: 6s - loss: 0.0044 - accuracy: 0.9994
 121/1688 [=>............................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9982
 133/1688 [=>............................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9984
 145/1688 [=>............................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9985
 157/1688 [=>............................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9982
 169/1688 [==>...........................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9983
 181/1688 [==>...........................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9981
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 205/1688 [==>...........................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9982
 217/1688 [==>...........................] - ETA: 6s - loss: 0.0058 - accuracy: 0.9983
 229/1688 [===>..........................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9984
 241/1688 [===>..........................] - ETA: 6s - loss: 0.0056 - accuracy: 0.9984
 253/1688 [===>..........................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9985
 265/1688 [===>..........................] - ETA: 6s - loss: 0.0053 - accuracy: 0.9986
 277/1688 [===>..........................] - ETA: 6s - loss: 0.0052 - accuracy: 0.9986
 289/1688 [====>.........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9987
 301/1688 [====>.........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9988
 313/1688 [====>.........................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9988
 325/1688 [====>.........................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9988
 337/1688 [====>.........................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9988
 349/1688 [=====>........................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9988
 361/1688 [=====>........................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 373/1688 [=====>........................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9986
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9986
 397/1688 [======>.......................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9986
 409/1688 [======>.......................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9986
 421/1688 [======>.......................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9986
 433/1688 [======>.......................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9986
 445/1688 [======>.......................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9987
 457/1688 [=======>......................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9987
 469/1688 [=======>......................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9987
 481/1688 [=======>......................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9988
 493/1688 [=======>......................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9987
 505/1688 [=======>......................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9987
 517/1688 [========>.....................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9987
 529/1688 [========>.....................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9986
 541/1688 [========>.....................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9987
 553/1688 [========>.....................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9986
 565/1688 [=========>....................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 577/1688 [=========>....................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 589/1688 [=========>....................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 601/1688 [=========>....................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 613/1688 [=========>....................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9987
 625/1688 [==========>...................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 637/1688 [==========>...................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 649/1688 [==========>...................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 661/1688 [==========>...................] - ETA: 4s - loss: 0.0050 - accuracy: 0.9987
 672/1688 [==========>...................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9987
 684/1688 [===========>..................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 696/1688 [===========>..................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9987
 708/1688 [===========>..................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9986
 720/1688 [===========>..................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9987
 732/1688 [============>.................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9987
 744/1688 [============>.................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9986
 756/1688 [============>.................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9986
 768/1688 [============>.................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9987
 780/1688 [============>.................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 792/1688 [=============>................] - ETA: 3s - loss: 0.0051 - accuracy: 0.9987
 804/1688 [=============>................] - ETA: 3s - loss: 0.0052 - accuracy: 0.9986
 816/1688 [=============>................] - ETA: 3s - loss: 0.0051 - accuracy: 0.9987
 828/1688 [=============>................] - ETA: 3s - loss: 0.0051 - accuracy: 0.9986
 840/1688 [=============>................] - ETA: 3s - loss: 0.0051 - accuracy: 0.9987
 852/1688 [==============>...............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9986
 864/1688 [==============>...............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9987
 876/1688 [==============>...............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9986
 888/1688 [==============>...............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9987
 900/1688 [==============>...............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9986
 912/1688 [===============>..............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9986
 924/1688 [===============>..............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9986
 936/1688 [===============>..............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9987
 948/1688 [===============>..............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9987
 960/1688 [================>.............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9987
 972/1688 [================>.............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9987
 984/1688 [================>.............] - ETA: 3s - loss: 0.0049 - accuracy: 0.9987
 996/1688 [================>.............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9987
1008/1688 [================>.............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9987
1020/1688 [=================>............] - ETA: 2s - loss: 0.0050 - accuracy: 0.9987
1032/1688 [=================>............] - ETA: 2s - loss: 0.0050 - accuracy: 0.9987
1044/1688 [=================>............] - ETA: 2s - loss: 0.0050 - accuracy: 0.9987
1056/1688 [=================>............] - ETA: 2s - loss: 0.0050 - accuracy: 0.9987
1068/1688 [=================>............] - ETA: 2s - loss: 0.0049 - accuracy: 0.9987
1080/1688 [==================>...........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1092/1688 [==================>...........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1104/1688 [==================>...........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1116/1688 [==================>...........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1128/1688 [===================>..........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1140/1688 [===================>..........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9988
1152/1688 [===================>..........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0050 - accuracy: 0.9988
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0050 - accuracy: 0.9988
1188/1688 [====================>.........] - ETA: 2s - loss: 0.0050 - accuracy: 0.9988
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0050 - accuracy: 0.9988
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0050 - accuracy: 0.9988
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0050 - accuracy: 0.9988
1236/1688 [====================>.........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0051 - accuracy: 0.9988
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1308/1688 [======================>.......] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1320/1688 [======================>.......] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0050 - accuracy: 0.9989
1380/1688 [=======================>......] - ETA: 1s - loss: 0.0049 - accuracy: 0.9989
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0049 - accuracy: 0.9989
1404/1688 [=======================>......] - ETA: 1s - loss: 0.0049 - accuracy: 0.9989
1416/1688 [========================>.....] - ETA: 1s - loss: 0.0049 - accuracy: 0.9989
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0049 - accuracy: 0.9988
1440/1688 [========================>.....] - ETA: 1s - loss: 0.0049 - accuracy: 0.9988
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0049 - accuracy: 0.9989
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0048 - accuracy: 0.9989
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0049 - accuracy: 0.9989
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0051 - accuracy: 0.9988
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0051 - accuracy: 0.9988
1643/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9988
1655/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9988
1667/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9988
1679/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9988
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0051 - accuracy: 0.9988 - val_loss: 0.0551 - val_accuracy: 0.9883
Epoch 5/5

   1/1688 [..............................] - ETA: 7s - loss: 0.0030 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0037 - accuracy: 1.0000
  25/1688 [..............................] - ETA: 7s - loss: 0.0038 - accuracy: 1.0000
  37/1688 [..............................] - ETA: 7s - loss: 0.0034 - accuracy: 1.0000
  49/1688 [..............................] - ETA: 7s - loss: 0.0030 - accuracy: 1.0000
  61/1688 [>.............................] - ETA: 7s - loss: 0.0029 - accuracy: 1.0000
  73/1688 [>.............................] - ETA: 7s - loss: 0.0031 - accuracy: 1.0000
  85/1688 [>.............................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9996
  97/1688 [>.............................] - ETA: 7s - loss: 0.0042 - accuracy: 0.9994
 109/1688 [>.............................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9994
 121/1688 [=>............................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9995
 133/1688 [=>............................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9995
 145/1688 [=>............................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9996
 157/1688 [=>............................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9996
 169/1688 [==>...........................] - ETA: 6s - loss: 0.0033 - accuracy: 0.9996
 181/1688 [==>...........................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9995
 192/1688 [==>...........................] - ETA: 6s - loss: 0.0034 - accuracy: 0.9995
 204/1688 [==>...........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9994
 216/1688 [==>...........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9994
 228/1688 [===>..........................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9992
 240/1688 [===>..........................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9992
 252/1688 [===>..........................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9993
 264/1688 [===>..........................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9993
 276/1688 [===>..........................] - ETA: 6s - loss: 0.0042 - accuracy: 0.9993
 288/1688 [====>.........................] - ETA: 6s - loss: 0.0041 - accuracy: 0.9993
 300/1688 [====>.........................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9994
 312/1688 [====>.........................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9994
 324/1688 [====>.........................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9994
 336/1688 [====>.........................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9994
 348/1688 [=====>........................] - ETA: 5s - loss: 0.0037 - accuracy: 0.9995
 360/1688 [=====>........................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9994
 372/1688 [=====>........................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9994
 384/1688 [=====>........................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9994
 396/1688 [======>.......................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9994
 408/1688 [======>.......................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9993
 420/1688 [======>.......................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9993
 432/1688 [======>.......................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9993
 444/1688 [======>.......................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9994
 456/1688 [=======>......................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9993
 468/1688 [=======>......................] - ETA: 5s - loss: 0.0042 - accuracy: 0.9993
 480/1688 [=======>......................] - ETA: 5s - loss: 0.0044 - accuracy: 0.9992
 492/1688 [=======>......................] - ETA: 5s - loss: 0.0044 - accuracy: 0.9992
 504/1688 [=======>......................] - ETA: 5s - loss: 0.0044 - accuracy: 0.9993
 516/1688 [========>.....................] - ETA: 5s - loss: 0.0044 - accuracy: 0.9993
 527/1688 [========>.....................] - ETA: 5s - loss: 0.0044 - accuracy: 0.9992
 539/1688 [========>.....................] - ETA: 5s - loss: 0.0045 - accuracy: 0.9992
 551/1688 [========>.....................] - ETA: 5s - loss: 0.0047 - accuracy: 0.9991
 563/1688 [=========>....................] - ETA: 5s - loss: 0.0046 - accuracy: 0.9992
 575/1688 [=========>....................] - ETA: 4s - loss: 0.0046 - accuracy: 0.9992
 587/1688 [=========>....................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9991
 599/1688 [=========>....................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9991
 611/1688 [=========>....................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9991
 623/1688 [==========>...................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9991
 635/1688 [==========>...................] - ETA: 4s - loss: 0.0046 - accuracy: 0.9991
 647/1688 [==========>...................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9991
 659/1688 [==========>...................] - ETA: 4s - loss: 0.0048 - accuracy: 0.9991
 671/1688 [==========>...................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9990
 683/1688 [===========>..................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9990
 695/1688 [===========>..................] - ETA: 4s - loss: 0.0046 - accuracy: 0.9991
 707/1688 [===========>..................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9990
 719/1688 [===========>..................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9990
 731/1688 [===========>..................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9990
 743/1688 [============>.................] - ETA: 4s - loss: 0.0047 - accuracy: 0.9990
 755/1688 [============>.................] - ETA: 4s - loss: 0.0046 - accuracy: 0.9990
 767/1688 [============>.................] - ETA: 4s - loss: 0.0046 - accuracy: 0.9991
 779/1688 [============>.................] - ETA: 4s - loss: 0.0046 - accuracy: 0.9991
 791/1688 [=============>................] - ETA: 3s - loss: 0.0046 - accuracy: 0.9991
 803/1688 [=============>................] - ETA: 3s - loss: 0.0046 - accuracy: 0.9990
 815/1688 [=============>................] - ETA: 3s - loss: 0.0046 - accuracy: 0.9990
 827/1688 [=============>................] - ETA: 3s - loss: 0.0046 - accuracy: 0.9991
 839/1688 [=============>................] - ETA: 3s - loss: 0.0045 - accuracy: 0.9991
 851/1688 [==============>...............] - ETA: 3s - loss: 0.0045 - accuracy: 0.9991
 863/1688 [==============>...............] - ETA: 3s - loss: 0.0045 - accuracy: 0.9991
 875/1688 [==============>...............] - ETA: 3s - loss: 0.0045 - accuracy: 0.9991
 887/1688 [==============>...............] - ETA: 3s - loss: 0.0044 - accuracy: 0.9991
 899/1688 [==============>...............] - ETA: 3s - loss: 0.0044 - accuracy: 0.9991
 911/1688 [===============>..............] - ETA: 3s - loss: 0.0044 - accuracy: 0.9991
 923/1688 [===============>..............] - ETA: 3s - loss: 0.0044 - accuracy: 0.9991
 935/1688 [===============>..............] - ETA: 3s - loss: 0.0044 - accuracy: 0.9991
 947/1688 [===============>..............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9991
 959/1688 [================>.............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9992
 971/1688 [================>.............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9992
 983/1688 [================>.............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9992
 995/1688 [================>.............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9992
1007/1688 [================>.............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
1019/1688 [=================>............] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1031/1688 [=================>............] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1043/1688 [=================>............] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1055/1688 [=================>............] - ETA: 2s - loss: 0.0043 - accuracy: 0.9992
1067/1688 [=================>............] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1079/1688 [==================>...........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1091/1688 [==================>...........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1103/1688 [==================>...........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1115/1688 [==================>...........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1127/1688 [===================>..........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1139/1688 [===================>..........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1151/1688 [===================>..........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9992
1163/1688 [===================>..........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1175/1688 [===================>..........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9991
1187/1688 [====================>.........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9992
1199/1688 [====================>.........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9992
1211/1688 [====================>.........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9991
1223/1688 [====================>.........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9992
1235/1688 [====================>.........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9992
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0042 - accuracy: 0.9992
1259/1688 [=====================>........] - ETA: 1s - loss: 0.0042 - accuracy: 0.9992
1271/1688 [=====================>........] - ETA: 1s - loss: 0.0042 - accuracy: 0.9992
1283/1688 [=====================>........] - ETA: 1s - loss: 0.0042 - accuracy: 0.9992
1295/1688 [======================>.......] - ETA: 1s - loss: 0.0042 - accuracy: 0.9992
1307/1688 [======================>.......] - ETA: 1s - loss: 0.0042 - accuracy: 0.9992
1319/1688 [======================>.......] - ETA: 1s - loss: 0.0042 - accuracy: 0.9992
1331/1688 [======================>.......] - ETA: 1s - loss: 0.0042 - accuracy: 0.9992
1343/1688 [======================>.......] - ETA: 1s - loss: 0.0043 - accuracy: 0.9992
1355/1688 [=======================>......] - ETA: 1s - loss: 0.0043 - accuracy: 0.9991
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0043 - accuracy: 0.9991
1379/1688 [=======================>......] - ETA: 1s - loss: 0.0043 - accuracy: 0.9991
1391/1688 [=======================>......] - ETA: 1s - loss: 0.0043 - accuracy: 0.9991
1403/1688 [=======================>......] - ETA: 1s - loss: 0.0044 - accuracy: 0.9991
1415/1688 [========================>.....] - ETA: 1s - loss: 0.0044 - accuracy: 0.9991
1427/1688 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9991
1439/1688 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9991
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9991
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1643/1688 [============================>.] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1655/1688 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1667/1688 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1679/1688 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0045 - accuracy: 0.9991 - val_loss: 0.0564 - val_accuracy: 0.9880

<keras.callbacks.History object at 0x7f7d220afd60>

score = model_quantized.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after fine tuning:', score)

Test accuracy after fine tuning: 0.9876000285148621

3. Convert

3.1 Convert to Akida model

When the quantized model produces satisfactory performance, it can be converted to the native Akida format. The convert function returns a model in Akida format ready for inference.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized)
model_akida.summary()

WARNING:tensorflow:5 out of the last 5 calls to <function NonTrackVariable.set_var at 0x7f7d0a26c1f0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 6 calls to <function NonTrackVariable.set_var at 0x7f7d0a26c700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          5
______________________________________________

__________________________________________________________________
Layer (type)                        Output shape  Kernel shape

=============== SW/conv2d-dequantizer_2 (Software) ===============

conv2d (InputConv2D)                [13, 13, 32]  (3, 3, 1, 32)
__________________________________________________________________
depthwise_conv2d (DepthwiseConv2D)  [7, 7, 32]    (3, 3, 32, 1)
__________________________________________________________________
conv2d_1 (Conv2D)                   [7, 7, 64]    (1, 1, 32, 64)
__________________________________________________________________
dense (Dense1D)                     [1, 1, 10]    (3136, 10)
__________________________________________________________________
dequantizer_2 (Dequantizer)         [1, 1, 10]    N/A
__________________________________________________________________

3.2. Check performance

accuracy = model_akida.evaluate(x_test, y_test)
print('Test accuracy after conversion:', accuracy)

# For non-regression purposes
assert accuracy > 0.96

Test accuracy after conversion: 0.9858999848365784

3.3 Show predictions for a single image

Display one of the test images, such as the first image in the dataset from above, to visualize the output of the model.

# Test a single example
sample_image = 0
image = x_test[sample_image]
outputs = model_akida.predict(image.reshape(1, 28, 28, 1))
print('Input Label: %i' % y_test[sample_image])

f, axarr = plt.subplots(1, 2)
axarr[0].imshow(x_test[sample_image].reshape((28, 28)), cmap=cm.Greys_r)
axarr[0].set_title('Class %d' % y_test[sample_image])
axarr[1].bar(range(10), outputs.squeeze())
axarr[1].set_xticks(range(10))
plt.show()

print(outputs.squeeze())

Input Label: 7
[-14.873109  -12.621015   -6.701727   -0.7304621 -16.686386  -10.731526
 -33.687523   11.017107   -7.3900304  -1.294306 ]

Consider the output from the model above. As is typical in backprop-trained models, the final layer is a Dense layer with one neuron for each of the 10 classes in the dataset. The goal of training is to maximize the response of the neuron corresponding to the label of each training sample while minimizing the responses of the other neurons.

In the bar chart above, you can see the outputs from all 10 neurons. It is easy to see that neuron 7 responds much more strongly than the others. The first sample is indeed a number 7.