Global Akida workflow

Using the MNIST dataset, this example shows the definition and training of a keras floating point model, its quantization to 8-bit with the help of calibration, its quantization to 4-bit using QAT and its conversion to Akida. Notice that the performance of the original keras floating point model is maintained throughout the Akida flow. Please refer to the Akida user guide for further information.

Note

Please refer to the TensorFlow tf.keras.models module for model creation/import details and the TensorFlow Guide for TensorFlow usage.

The MNIST example below is light enough so that a GPU is not needed for training.

Akida workflow

1. Create and train

1.1. Load and reshape MNIST dataset

import numpy as np

import matplotlib.cm as cm
import matplotlib.pyplot as plt

from keras.datasets import mnist

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Add a channels dimension to the image sets as Akida expects 4-D inputs (corresponding to
# (num_samples, width, height, channels). Note: MNIST is a grayscale dataset and is unusual
# in this respect - most image data already includes a channel dimension, and this step will
# not be necessary.
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)

# Display a few images from the test set
f, axarr = plt.subplots(1, 4)
for i in range(0, 4):
    axarr[i].imshow(x_test[i].reshape((28, 28)), cmap=cm.Greys_r)
    axarr[i].set_title('Class %d' % y_test[i])
plt.show()

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz

    8192/11490434 [..............................] - ETA: 0s
   24576/11490434 [..............................] - ETA: 1:14
   40960/11490434 [..............................] - ETA: 1:02
   57344/11490434 [..............................] - ETA: 1:00
   81920/11490434 [..............................] - ETA: 51s 
  106496/11490434 [..............................] - ETA: 45s
  139264/11490434 [..............................] - ETA: 39s
  180224/11490434 [..............................] - ETA: 33s
  229376/11490434 [..............................] - ETA: 29s
  286720/11490434 [..............................] - ETA: 25s
  360448/11490434 [..............................] - ETA: 21s
  458752/11490434 [>.............................] - ETA: 18s
  581632/11490434 [>.............................] - ETA: 15s
  729088/11490434 [>.............................] - ETA: 12s
  917504/11490434 [=>............................] - ETA: 10s
 1146880/11490434 [=>............................] - ETA: 8s 
 1458176/11490434 [==>...........................] - ETA: 6s
 1851392/11490434 [===>..........................] - ETA: 5s
 2260992/11490434 [====>.........................] - ETA: 4s
 2834432/11490434 [======>.......................] - ETA: 3s
 3096576/11490434 [=======>......................] - ETA: 3s
 3981312/11490434 [=========>....................] - ETA: 2s
 4898816/11490434 [===========>..................] - ETA: 1s
 6029312/11490434 [==============>...............] - ETA: 1s
 6258688/11490434 [===============>..............] - ETA: 1s
 6832128/11490434 [================>.............] - ETA: 1s
 7405568/11490434 [==================>...........] - ETA: 0s
 7733248/11490434 [===================>..........] - ETA: 0s
 8290304/11490434 [====================>.........] - ETA: 0s
 8863744/11490434 [======================>.......] - ETA: 0s
 9224192/11490434 [=======================>......] - ETA: 0s
 9797632/11490434 [========================>.....] - ETA: 0s
10371072/11490434 [==========================>...] - ETA: 0s
10715136/11490434 [==========================>...] - ETA: 0s
11288576/11490434 [============================>.] - ETA: 0s
11490434/11490434 [==============================] - 2s 0us/step

1.2. Model definition

Note that at this stage, there is nothing specific to the Akida IP. The model constructed below, as inspired by this example, is a completely standard Keras CNN model.

import keras

model_keras = keras.models.Sequential([
    keras.layers.Rescaling(1. / 255, input_shape=(28, 28, 1)),
    keras.layers.Conv2D(filters=32, kernel_size=3, strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    # Separable layer
    keras.layers.DepthwiseConv2D(kernel_size=3, padding='same', strides=2),
    keras.layers.Conv2D(filters=64, kernel_size=1, padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()

Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling (Rescaling)       (None, 28, 28, 1)         0

 conv2d (Conv2D)             (None, 13, 13, 32)        320

 batch_normalization (BatchN  (None, 13, 13, 32)       128
 ormalization)

 re_lu (ReLU)                (None, 13, 13, 32)        0

 depthwise_conv2d (Depthwise  (None, 7, 7, 32)         320
 Conv2D)

 conv2d_1 (Conv2D)           (None, 7, 7, 64)          2112

 batch_normalization_1 (Batc  (None, 7, 7, 64)         256
 hNormalization)

 re_lu_1 (ReLU)              (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (Dense)               (None, 10)                31370

=================================================================
Total params: 34,506
Trainable params: 34,314
Non-trainable params: 192
_________________________________________________________________

1.3. Model training

Given the model created above, train the model and check its accuracy. The model should achieve a test accuracy over 98% after 10 epochs.

from keras.optimizers import Adam

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-3),
    metrics=['accuracy'])

_ = model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)

Epoch 1/10

   1/1688 [..............................] - ETA: 42:15 - loss: 2.7653 - accuracy: 0.0000e+00
  31/1688 [..............................] - ETA: 2s - loss: 1.1709 - accuracy: 0.6321       
  60/1688 [>.............................] - ETA: 2s - loss: 0.8270 - accuracy: 0.7469
  90/1688 [>.............................] - ETA: 2s - loss: 0.6889 - accuracy: 0.7917
 120/1688 [=>............................] - ETA: 2s - loss: 0.6008 - accuracy: 0.8201
 150/1688 [=>............................] - ETA: 2s - loss: 0.5376 - accuracy: 0.8398
 180/1688 [==>...........................] - ETA: 2s - loss: 0.4911 - accuracy: 0.8536
 210/1688 [==>...........................] - ETA: 2s - loss: 0.4621 - accuracy: 0.8626
 241/1688 [===>..........................] - ETA: 2s - loss: 0.4286 - accuracy: 0.8729
 271/1688 [===>..........................] - ETA: 2s - loss: 0.4086 - accuracy: 0.8794
 302/1688 [====>.........................] - ETA: 2s - loss: 0.3845 - accuracy: 0.8863
 332/1688 [====>.........................] - ETA: 2s - loss: 0.3688 - accuracy: 0.8913
 362/1688 [=====>........................] - ETA: 2s - loss: 0.3524 - accuracy: 0.8962
 393/1688 [=====>........................] - ETA: 2s - loss: 0.3387 - accuracy: 0.8997
 422/1688 [======>.......................] - ETA: 2s - loss: 0.3242 - accuracy: 0.9039
 452/1688 [=======>......................] - ETA: 2s - loss: 0.3102 - accuracy: 0.9079
 483/1688 [=======>......................] - ETA: 2s - loss: 0.2985 - accuracy: 0.9114
 513/1688 [========>.....................] - ETA: 1s - loss: 0.2885 - accuracy: 0.9147
 544/1688 [========>.....................] - ETA: 1s - loss: 0.2785 - accuracy: 0.9176
 574/1688 [=========>....................] - ETA: 1s - loss: 0.2698 - accuracy: 0.9199
 604/1688 [=========>....................] - ETA: 1s - loss: 0.2634 - accuracy: 0.9216
 634/1688 [==========>...................] - ETA: 1s - loss: 0.2566 - accuracy: 0.9234
 664/1688 [==========>...................] - ETA: 1s - loss: 0.2529 - accuracy: 0.9250
 696/1688 [===========>..................] - ETA: 1s - loss: 0.2485 - accuracy: 0.9261
 727/1688 [===========>..................] - ETA: 1s - loss: 0.2447 - accuracy: 0.9273
 756/1688 [============>.................] - ETA: 1s - loss: 0.2410 - accuracy: 0.9282
 786/1688 [============>.................] - ETA: 1s - loss: 0.2358 - accuracy: 0.9297
 816/1688 [=============>................] - ETA: 1s - loss: 0.2309 - accuracy: 0.9310
 845/1688 [==============>...............] - ETA: 1s - loss: 0.2273 - accuracy: 0.9319
 875/1688 [==============>...............] - ETA: 1s - loss: 0.2225 - accuracy: 0.9333
 905/1688 [===============>..............] - ETA: 1s - loss: 0.2180 - accuracy: 0.9346
 936/1688 [===============>..............] - ETA: 1s - loss: 0.2141 - accuracy: 0.9359
 966/1688 [================>.............] - ETA: 1s - loss: 0.2106 - accuracy: 0.9369
 997/1688 [================>.............] - ETA: 1s - loss: 0.2084 - accuracy: 0.9376
1027/1688 [=================>............] - ETA: 1s - loss: 0.2048 - accuracy: 0.9386
1057/1688 [=================>............] - ETA: 1s - loss: 0.2026 - accuracy: 0.9392
1087/1688 [==================>...........] - ETA: 1s - loss: 0.1998 - accuracy: 0.9400
1117/1688 [==================>...........] - ETA: 0s - loss: 0.1976 - accuracy: 0.9407
1147/1688 [===================>..........] - ETA: 0s - loss: 0.1961 - accuracy: 0.9413
1177/1688 [===================>..........] - ETA: 0s - loss: 0.1935 - accuracy: 0.9419
1208/1688 [====================>.........] - ETA: 0s - loss: 0.1919 - accuracy: 0.9425
1238/1688 [=====================>........] - ETA: 0s - loss: 0.1899 - accuracy: 0.9432
1268/1688 [=====================>........] - ETA: 0s - loss: 0.1883 - accuracy: 0.9437
1298/1688 [======================>.......] - ETA: 0s - loss: 0.1864 - accuracy: 0.9443
1328/1688 [======================>.......] - ETA: 0s - loss: 0.1848 - accuracy: 0.9448
1358/1688 [=======================>......] - ETA: 0s - loss: 0.1820 - accuracy: 0.9456
1389/1688 [=======================>......] - ETA: 0s - loss: 0.1798 - accuracy: 0.9464
1419/1688 [========================>.....] - ETA: 0s - loss: 0.1778 - accuracy: 0.9469
1449/1688 [========================>.....] - ETA: 0s - loss: 0.1759 - accuracy: 0.9474
1480/1688 [=========================>....] - ETA: 0s - loss: 0.1741 - accuracy: 0.9478
1511/1688 [=========================>....] - ETA: 0s - loss: 0.1720 - accuracy: 0.9485
1541/1688 [==========================>...] - ETA: 0s - loss: 0.1702 - accuracy: 0.9490
1571/1688 [==========================>...] - ETA: 0s - loss: 0.1690 - accuracy: 0.9493
1601/1688 [===========================>..] - ETA: 0s - loss: 0.1666 - accuracy: 0.9500
1631/1688 [===========================>..] - ETA: 0s - loss: 0.1655 - accuracy: 0.9504
1663/1688 [============================>.] - ETA: 0s - loss: 0.1638 - accuracy: 0.9509
1688/1688 [==============================] - ETA: 0s - loss: 0.1630 - accuracy: 0.9512
1688/1688 [==============================] - 5s 2ms/step - loss: 0.1630 - accuracy: 0.9512 - val_loss: 0.0752 - val_accuracy: 0.9805
Epoch 2/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0110 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0417 - accuracy: 0.9834
  62/1688 [>.............................] - ETA: 2s - loss: 0.0487 - accuracy: 0.9829
  92/1688 [>.............................] - ETA: 2s - loss: 0.0605 - accuracy: 0.9800
 122/1688 [=>............................] - ETA: 2s - loss: 0.0628 - accuracy: 0.9805
 152/1688 [=>............................] - ETA: 2s - loss: 0.0610 - accuracy: 0.9799
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0576 - accuracy: 0.9811
 212/1688 [==>...........................] - ETA: 2s - loss: 0.0558 - accuracy: 0.9819
 242/1688 [===>..........................] - ETA: 2s - loss: 0.0578 - accuracy: 0.9808
 273/1688 [===>..........................] - ETA: 2s - loss: 0.0613 - accuracy: 0.9796
 303/1688 [====>.........................] - ETA: 2s - loss: 0.0629 - accuracy: 0.9793
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0638 - accuracy: 0.9789
 364/1688 [=====>........................] - ETA: 2s - loss: 0.0643 - accuracy: 0.9787
 395/1688 [======>.......................] - ETA: 2s - loss: 0.0632 - accuracy: 0.9789
 425/1688 [======>.......................] - ETA: 2s - loss: 0.0640 - accuracy: 0.9787
 455/1688 [=======>......................] - ETA: 2s - loss: 0.0660 - accuracy: 0.9782
 484/1688 [=======>......................] - ETA: 2s - loss: 0.0647 - accuracy: 0.9786
 515/1688 [========>.....................] - ETA: 1s - loss: 0.0655 - accuracy: 0.9785
 544/1688 [========>.....................] - ETA: 1s - loss: 0.0658 - accuracy: 0.9783
 574/1688 [=========>....................] - ETA: 1s - loss: 0.0653 - accuracy: 0.9784
 604/1688 [=========>....................] - ETA: 1s - loss: 0.0663 - accuracy: 0.9784
 635/1688 [==========>...................] - ETA: 1s - loss: 0.0689 - accuracy: 0.9777
 665/1688 [==========>...................] - ETA: 1s - loss: 0.0689 - accuracy: 0.9779
 695/1688 [===========>..................] - ETA: 1s - loss: 0.0687 - accuracy: 0.9780
 726/1688 [===========>..................] - ETA: 1s - loss: 0.0708 - accuracy: 0.9775
 756/1688 [============>.................] - ETA: 1s - loss: 0.0706 - accuracy: 0.9778
 787/1688 [============>.................] - ETA: 1s - loss: 0.0702 - accuracy: 0.9779
 817/1688 [=============>................] - ETA: 1s - loss: 0.0698 - accuracy: 0.9779
 846/1688 [==============>...............] - ETA: 1s - loss: 0.0696 - accuracy: 0.9780
 876/1688 [==============>...............] - ETA: 1s - loss: 0.0703 - accuracy: 0.9778
 905/1688 [===============>..............] - ETA: 1s - loss: 0.0705 - accuracy: 0.9779
 936/1688 [===============>..............] - ETA: 1s - loss: 0.0704 - accuracy: 0.9778
 967/1688 [================>.............] - ETA: 1s - loss: 0.0700 - accuracy: 0.9780
 997/1688 [================>.............] - ETA: 1s - loss: 0.0691 - accuracy: 0.9782
1027/1688 [=================>............] - ETA: 1s - loss: 0.0686 - accuracy: 0.9784
1057/1688 [=================>............] - ETA: 1s - loss: 0.0685 - accuracy: 0.9786
1088/1688 [==================>...........] - ETA: 1s - loss: 0.0680 - accuracy: 0.9788
1118/1688 [==================>...........] - ETA: 0s - loss: 0.0677 - accuracy: 0.9788
1149/1688 [===================>..........] - ETA: 0s - loss: 0.0675 - accuracy: 0.9787
1180/1688 [===================>..........] - ETA: 0s - loss: 0.0670 - accuracy: 0.9789
1211/1688 [====================>.........] - ETA: 0s - loss: 0.0670 - accuracy: 0.9789
1241/1688 [=====================>........] - ETA: 0s - loss: 0.0666 - accuracy: 0.9789
1271/1688 [=====================>........] - ETA: 0s - loss: 0.0669 - accuracy: 0.9789
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0678 - accuracy: 0.9788
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0679 - accuracy: 0.9788
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0674 - accuracy: 0.9789
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0672 - accuracy: 0.9789
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0671 - accuracy: 0.9788
1455/1688 [========================>.....] - ETA: 0s - loss: 0.0671 - accuracy: 0.9789
1486/1688 [=========================>....] - ETA: 0s - loss: 0.0664 - accuracy: 0.9791
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0665 - accuracy: 0.9791
1546/1688 [==========================>...] - ETA: 0s - loss: 0.0666 - accuracy: 0.9791
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0665 - accuracy: 0.9791
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0664 - accuracy: 0.9792
1639/1688 [============================>.] - ETA: 0s - loss: 0.0663 - accuracy: 0.9793
1670/1688 [============================>.] - ETA: 0s - loss: 0.0663 - accuracy: 0.9793
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0664 - accuracy: 0.9792 - val_loss: 0.0875 - val_accuracy: 0.9753
Epoch 3/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0244 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0357 - accuracy: 0.9848
  63/1688 [>.............................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9836
  93/1688 [>.............................] - ETA: 2s - loss: 0.0439 - accuracy: 0.9845
 124/1688 [=>............................] - ETA: 2s - loss: 0.0444 - accuracy: 0.9846
 155/1688 [=>............................] - ETA: 2s - loss: 0.0460 - accuracy: 0.9845
 185/1688 [==>...........................] - ETA: 2s - loss: 0.0441 - accuracy: 0.9848
 215/1688 [==>...........................] - ETA: 2s - loss: 0.0453 - accuracy: 0.9846
 245/1688 [===>..........................] - ETA: 2s - loss: 0.0462 - accuracy: 0.9849
 276/1688 [===>..........................] - ETA: 2s - loss: 0.0460 - accuracy: 0.9848
 307/1688 [====>.........................] - ETA: 2s - loss: 0.0479 - accuracy: 0.9843
 338/1688 [=====>........................] - ETA: 2s - loss: 0.0474 - accuracy: 0.9848
 369/1688 [=====>........................] - ETA: 2s - loss: 0.0466 - accuracy: 0.9849
 400/1688 [======>.......................] - ETA: 2s - loss: 0.0475 - accuracy: 0.9848
 430/1688 [======>.......................] - ETA: 2s - loss: 0.0478 - accuracy: 0.9847
 460/1688 [=======>......................] - ETA: 2s - loss: 0.0483 - accuracy: 0.9844
 490/1688 [=======>......................] - ETA: 1s - loss: 0.0485 - accuracy: 0.9847
 520/1688 [========>.....................] - ETA: 1s - loss: 0.0486 - accuracy: 0.9847
 550/1688 [========>.....................] - ETA: 1s - loss: 0.0482 - accuracy: 0.9849
 581/1688 [=========>....................] - ETA: 1s - loss: 0.0470 - accuracy: 0.9852
 612/1688 [=========>....................] - ETA: 1s - loss: 0.0468 - accuracy: 0.9850
 642/1688 [==========>...................] - ETA: 1s - loss: 0.0474 - accuracy: 0.9850
 672/1688 [==========>...................] - ETA: 1s - loss: 0.0481 - accuracy: 0.9848
 702/1688 [===========>..................] - ETA: 1s - loss: 0.0485 - accuracy: 0.9848
 732/1688 [============>.................] - ETA: 1s - loss: 0.0484 - accuracy: 0.9848
 762/1688 [============>.................] - ETA: 1s - loss: 0.0480 - accuracy: 0.9848
 792/1688 [=============>................] - ETA: 1s - loss: 0.0481 - accuracy: 0.9849
 822/1688 [=============>................] - ETA: 1s - loss: 0.0484 - accuracy: 0.9848
 852/1688 [==============>...............] - ETA: 1s - loss: 0.0487 - accuracy: 0.9847
 883/1688 [==============>...............] - ETA: 1s - loss: 0.0488 - accuracy: 0.9846
 914/1688 [===============>..............] - ETA: 1s - loss: 0.0484 - accuracy: 0.9846
 944/1688 [===============>..............] - ETA: 1s - loss: 0.0495 - accuracy: 0.9844
 975/1688 [================>.............] - ETA: 1s - loss: 0.0495 - accuracy: 0.9845
1005/1688 [================>.............] - ETA: 1s - loss: 0.0497 - accuracy: 0.9844
1035/1688 [=================>............] - ETA: 1s - loss: 0.0494 - accuracy: 0.9844
1065/1688 [=================>............] - ETA: 1s - loss: 0.0494 - accuracy: 0.9844
1095/1688 [==================>...........] - ETA: 0s - loss: 0.0490 - accuracy: 0.9845
1125/1688 [==================>...........] - ETA: 0s - loss: 0.0492 - accuracy: 0.9844
1156/1688 [===================>..........] - ETA: 0s - loss: 0.0499 - accuracy: 0.9842
1187/1688 [====================>.........] - ETA: 0s - loss: 0.0506 - accuracy: 0.9840
1217/1688 [====================>.........] - ETA: 0s - loss: 0.0504 - accuracy: 0.9841
1248/1688 [=====================>........] - ETA: 0s - loss: 0.0510 - accuracy: 0.9839
1279/1688 [=====================>........] - ETA: 0s - loss: 0.0505 - accuracy: 0.9841
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0503 - accuracy: 0.9842
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0505 - accuracy: 0.9842
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0502 - accuracy: 0.9843
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0499 - accuracy: 0.9844
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0501 - accuracy: 0.9843
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0501 - accuracy: 0.9842
1489/1688 [=========================>....] - ETA: 0s - loss: 0.0497 - accuracy: 0.9843
1520/1688 [==========================>...] - ETA: 0s - loss: 0.0498 - accuracy: 0.9843
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0498 - accuracy: 0.9843
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0496 - accuracy: 0.9843
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0503 - accuracy: 0.9841
1642/1688 [============================>.] - ETA: 0s - loss: 0.0503 - accuracy: 0.9841
1672/1688 [============================>.] - ETA: 0s - loss: 0.0503 - accuracy: 0.9840
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0502 - accuracy: 0.9841 - val_loss: 0.0676 - val_accuracy: 0.9813
Epoch 4/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0075 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0411 - accuracy: 0.9820
  63/1688 [>.............................] - ETA: 2s - loss: 0.0344 - accuracy: 0.9866
  93/1688 [>.............................] - ETA: 2s - loss: 0.0344 - accuracy: 0.9869
 123/1688 [=>............................] - ETA: 2s - loss: 0.0380 - accuracy: 0.9850
 153/1688 [=>............................] - ETA: 2s - loss: 0.0356 - accuracy: 0.9859
 183/1688 [==>...........................] - ETA: 2s - loss: 0.0336 - accuracy: 0.9865
 214/1688 [==>...........................] - ETA: 2s - loss: 0.0312 - accuracy: 0.9876
 244/1688 [===>..........................] - ETA: 2s - loss: 0.0327 - accuracy: 0.9872
 274/1688 [===>..........................] - ETA: 2s - loss: 0.0333 - accuracy: 0.9872
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0342 - accuracy: 0.9873
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0350 - accuracy: 0.9873
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0363 - accuracy: 0.9870
 396/1688 [======>.......................] - ETA: 2s - loss: 0.0360 - accuracy: 0.9872
 426/1688 [======>.......................] - ETA: 2s - loss: 0.0353 - accuracy: 0.9873
 456/1688 [=======>......................] - ETA: 2s - loss: 0.0358 - accuracy: 0.9874
 486/1688 [=======>......................] - ETA: 2s - loss: 0.0366 - accuracy: 0.9869
 517/1688 [========>.....................] - ETA: 1s - loss: 0.0370 - accuracy: 0.9870
 547/1688 [========>.....................] - ETA: 1s - loss: 0.0364 - accuracy: 0.9871
 578/1688 [=========>....................] - ETA: 1s - loss: 0.0358 - accuracy: 0.9874
 609/1688 [=========>....................] - ETA: 1s - loss: 0.0365 - accuracy: 0.9872
 640/1688 [==========>...................] - ETA: 1s - loss: 0.0365 - accuracy: 0.9872
 670/1688 [==========>...................] - ETA: 1s - loss: 0.0366 - accuracy: 0.9872
 700/1688 [===========>..................] - ETA: 1s - loss: 0.0368 - accuracy: 0.9872
 730/1688 [===========>..................] - ETA: 1s - loss: 0.0374 - accuracy: 0.9869
 760/1688 [============>.................] - ETA: 1s - loss: 0.0371 - accuracy: 0.9870
 791/1688 [=============>................] - ETA: 1s - loss: 0.0378 - accuracy: 0.9868
 822/1688 [=============>................] - ETA: 1s - loss: 0.0382 - accuracy: 0.9869
 852/1688 [==============>...............] - ETA: 1s - loss: 0.0382 - accuracy: 0.9868
 882/1688 [==============>...............] - ETA: 1s - loss: 0.0386 - accuracy: 0.9867
 912/1688 [===============>..............] - ETA: 1s - loss: 0.0400 - accuracy: 0.9863
 942/1688 [===============>..............] - ETA: 1s - loss: 0.0401 - accuracy: 0.9863
 972/1688 [================>.............] - ETA: 1s - loss: 0.0400 - accuracy: 0.9864
1004/1688 [================>.............] - ETA: 1s - loss: 0.0396 - accuracy: 0.9866
1034/1688 [=================>............] - ETA: 1s - loss: 0.0397 - accuracy: 0.9865
1064/1688 [=================>............] - ETA: 1s - loss: 0.0393 - accuracy: 0.9867
1094/1688 [==================>...........] - ETA: 0s - loss: 0.0398 - accuracy: 0.9865
1124/1688 [==================>...........] - ETA: 0s - loss: 0.0394 - accuracy: 0.9867
1154/1688 [===================>..........] - ETA: 0s - loss: 0.0392 - accuracy: 0.9868
1184/1688 [====================>.........] - ETA: 0s - loss: 0.0392 - accuracy: 0.9867
1215/1688 [====================>.........] - ETA: 0s - loss: 0.0391 - accuracy: 0.9867
1245/1688 [=====================>........] - ETA: 0s - loss: 0.0390 - accuracy: 0.9867
1275/1688 [=====================>........] - ETA: 0s - loss: 0.0394 - accuracy: 0.9867
1305/1688 [======================>.......] - ETA: 0s - loss: 0.0393 - accuracy: 0.9866
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0396 - accuracy: 0.9866
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0395 - accuracy: 0.9866
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0394 - accuracy: 0.9866
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0395 - accuracy: 0.9865
1460/1688 [========================>.....] - ETA: 0s - loss: 0.0391 - accuracy: 0.9866
1491/1688 [=========================>....] - ETA: 0s - loss: 0.0396 - accuracy: 0.9865
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0397 - accuracy: 0.9864
1552/1688 [==========================>...] - ETA: 0s - loss: 0.0399 - accuracy: 0.9864
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0399 - accuracy: 0.9864
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0402 - accuracy: 0.9862
1642/1688 [============================>.] - ETA: 0s - loss: 0.0404 - accuracy: 0.9862
1672/1688 [============================>.] - ETA: 0s - loss: 0.0402 - accuracy: 0.9863
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0404 - accuracy: 0.9862 - val_loss: 0.0597 - val_accuracy: 0.9842
Epoch 5/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0339 - accuracy: 1.0000
  35/1688 [..............................] - ETA: 2s - loss: 0.0252 - accuracy: 0.9902
  65/1688 [>.............................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9913
  95/1688 [>.............................] - ETA: 2s - loss: 0.0224 - accuracy: 0.9928
 126/1688 [=>............................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9936
 156/1688 [=>............................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9932
 186/1688 [==>...........................] - ETA: 2s - loss: 0.0248 - accuracy: 0.9926
 217/1688 [==>...........................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9915
 247/1688 [===>..........................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9916
 277/1688 [===>..........................] - ETA: 2s - loss: 0.0262 - accuracy: 0.9915
 308/1688 [====>.........................] - ETA: 2s - loss: 0.0272 - accuracy: 0.9911
 337/1688 [====>.........................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9910
 364/1688 [=====>........................] - ETA: 2s - loss: 0.0273 - accuracy: 0.9910
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9910
 424/1688 [======>.......................] - ETA: 2s - loss: 0.0269 - accuracy: 0.9909
 453/1688 [=======>......................] - ETA: 2s - loss: 0.0272 - accuracy: 0.9909
 482/1688 [=======>......................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9909
 512/1688 [========>.....................] - ETA: 1s - loss: 0.0279 - accuracy: 0.9910
 543/1688 [========>.....................] - ETA: 1s - loss: 0.0283 - accuracy: 0.9907
 574/1688 [=========>....................] - ETA: 1s - loss: 0.0287 - accuracy: 0.9906
 605/1688 [=========>....................] - ETA: 1s - loss: 0.0283 - accuracy: 0.9908
 636/1688 [==========>...................] - ETA: 1s - loss: 0.0291 - accuracy: 0.9906
 667/1688 [==========>...................] - ETA: 1s - loss: 0.0292 - accuracy: 0.9906
 697/1688 [===========>..................] - ETA: 1s - loss: 0.0291 - accuracy: 0.9907
 727/1688 [===========>..................] - ETA: 1s - loss: 0.0297 - accuracy: 0.9903
 758/1688 [============>.................] - ETA: 1s - loss: 0.0306 - accuracy: 0.9900
 789/1688 [=============>................] - ETA: 1s - loss: 0.0309 - accuracy: 0.9899
 819/1688 [=============>................] - ETA: 1s - loss: 0.0307 - accuracy: 0.9900
 849/1688 [==============>...............] - ETA: 1s - loss: 0.0305 - accuracy: 0.9900
 880/1688 [==============>...............] - ETA: 1s - loss: 0.0299 - accuracy: 0.9903
 910/1688 [===============>..............] - ETA: 1s - loss: 0.0299 - accuracy: 0.9902
 941/1688 [===============>..............] - ETA: 1s - loss: 0.0305 - accuracy: 0.9900
 970/1688 [================>.............] - ETA: 1s - loss: 0.0308 - accuracy: 0.9899
1000/1688 [================>.............] - ETA: 1s - loss: 0.0307 - accuracy: 0.9899
1030/1688 [=================>............] - ETA: 1s - loss: 0.0314 - accuracy: 0.9898
1060/1688 [=================>............] - ETA: 1s - loss: 0.0309 - accuracy: 0.9899
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0310 - accuracy: 0.9899
1121/1688 [==================>...........] - ETA: 0s - loss: 0.0311 - accuracy: 0.9897
1152/1688 [===================>..........] - ETA: 0s - loss: 0.0313 - accuracy: 0.9897
1183/1688 [====================>.........] - ETA: 0s - loss: 0.0316 - accuracy: 0.9896
1214/1688 [====================>.........] - ETA: 0s - loss: 0.0314 - accuracy: 0.9896
1244/1688 [=====================>........] - ETA: 0s - loss: 0.0317 - accuracy: 0.9895
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0318 - accuracy: 0.9895
1305/1688 [======================>.......] - ETA: 0s - loss: 0.0319 - accuracy: 0.9894
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0320 - accuracy: 0.9893
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0320 - accuracy: 0.9894
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0326 - accuracy: 0.9892
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0329 - accuracy: 0.9891
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0328 - accuracy: 0.9892
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0327 - accuracy: 0.9892
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0327 - accuracy: 0.9891
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0329 - accuracy: 0.9890
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0330 - accuracy: 0.9890
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0330 - accuracy: 0.9890
1639/1688 [============================>.] - ETA: 0s - loss: 0.0329 - accuracy: 0.9891
1669/1688 [============================>.] - ETA: 0s - loss: 0.0330 - accuracy: 0.9891
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0330 - accuracy: 0.9890 - val_loss: 0.0647 - val_accuracy: 0.9838
Epoch 6/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0025 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9915
  63/1688 [>.............................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9936
  92/1688 [>.............................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9939
 122/1688 [=>............................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9936
 153/1688 [=>............................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9937
 183/1688 [==>...........................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9933
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9935
 243/1688 [===>..........................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9933
 273/1688 [===>..........................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9924
 304/1688 [====>.........................] - ETA: 2s - loss: 0.0224 - accuracy: 0.9927
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9923
 364/1688 [=====>........................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9922
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9921
 425/1688 [======>.......................] - ETA: 2s - loss: 0.0225 - accuracy: 0.9924
 455/1688 [=======>......................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9927
 486/1688 [=======>......................] - ETA: 2s - loss: 0.0223 - accuracy: 0.9925
 516/1688 [========>.....................] - ETA: 1s - loss: 0.0224 - accuracy: 0.9926
 547/1688 [========>.....................] - ETA: 1s - loss: 0.0224 - accuracy: 0.9926
 577/1688 [=========>....................] - ETA: 1s - loss: 0.0227 - accuracy: 0.9926
 608/1688 [=========>....................] - ETA: 1s - loss: 0.0224 - accuracy: 0.9926
 638/1688 [==========>...................] - ETA: 1s - loss: 0.0235 - accuracy: 0.9921
 668/1688 [==========>...................] - ETA: 1s - loss: 0.0238 - accuracy: 0.9922
 698/1688 [===========>..................] - ETA: 1s - loss: 0.0239 - accuracy: 0.9923
 729/1688 [===========>..................] - ETA: 1s - loss: 0.0238 - accuracy: 0.9924
 759/1688 [============>.................] - ETA: 1s - loss: 0.0235 - accuracy: 0.9925
 789/1688 [=============>................] - ETA: 1s - loss: 0.0231 - accuracy: 0.9925
 820/1688 [=============>................] - ETA: 1s - loss: 0.0235 - accuracy: 0.9923
 851/1688 [==============>...............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9922
 882/1688 [==============>...............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9923
 913/1688 [===============>..............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9924
 944/1688 [===============>..............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9924
 974/1688 [================>.............] - ETA: 1s - loss: 0.0230 - accuracy: 0.9923
1004/1688 [================>.............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9922
1035/1688 [=================>............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9921
1066/1688 [=================>............] - ETA: 1s - loss: 0.0238 - accuracy: 0.9919
1096/1688 [==================>...........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9919
1126/1688 [===================>..........] - ETA: 0s - loss: 0.0241 - accuracy: 0.9917
1157/1688 [===================>..........] - ETA: 0s - loss: 0.0245 - accuracy: 0.9917
1186/1688 [====================>.........] - ETA: 0s - loss: 0.0246 - accuracy: 0.9916
1216/1688 [====================>.........] - ETA: 0s - loss: 0.0250 - accuracy: 0.9916
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0252 - accuracy: 0.9915
1277/1688 [=====================>........] - ETA: 0s - loss: 0.0251 - accuracy: 0.9916
1307/1688 [======================>.......] - ETA: 0s - loss: 0.0253 - accuracy: 0.9916
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0251 - accuracy: 0.9916
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0253 - accuracy: 0.9915
1397/1688 [=======================>......] - ETA: 0s - loss: 0.0252 - accuracy: 0.9915
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0254 - accuracy: 0.9914
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0257 - accuracy: 0.9914
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0253 - accuracy: 0.9914
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0257 - accuracy: 0.9914
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0257 - accuracy: 0.9914
1638/1688 [============================>.] - ETA: 0s - loss: 0.0259 - accuracy: 0.9914
1668/1688 [============================>.] - ETA: 0s - loss: 0.0264 - accuracy: 0.9912
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0268 - accuracy: 0.9911 - val_loss: 0.0665 - val_accuracy: 0.9833
Epoch 7/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0103 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9971
  62/1688 [>.............................] - ETA: 2s - loss: 0.0123 - accuracy: 0.9970
  92/1688 [>.............................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9959
 122/1688 [=>............................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9954
 153/1688 [=>............................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9957
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9949
 214/1688 [==>...........................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9946
 244/1688 [===>..........................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9944
 275/1688 [===>..........................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9943
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9944
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9945
 365/1688 [=====>........................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9948
 395/1688 [======>.......................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9947
 426/1688 [======>.......................] - ETA: 2s - loss: 0.0169 - accuracy: 0.9945
 456/1688 [=======>......................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9944
 486/1688 [=======>......................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9941
 516/1688 [========>.....................] - ETA: 1s - loss: 0.0182 - accuracy: 0.9941
 546/1688 [========>.....................] - ETA: 1s - loss: 0.0183 - accuracy: 0.9941
 576/1688 [=========>....................] - ETA: 1s - loss: 0.0185 - accuracy: 0.9940
 606/1688 [=========>....................] - ETA: 1s - loss: 0.0185 - accuracy: 0.9938
 636/1688 [==========>...................] - ETA: 1s - loss: 0.0195 - accuracy: 0.9934
 666/1688 [==========>...................] - ETA: 1s - loss: 0.0198 - accuracy: 0.9931
 696/1688 [===========>..................] - ETA: 1s - loss: 0.0198 - accuracy: 0.9931
 727/1688 [===========>..................] - ETA: 1s - loss: 0.0203 - accuracy: 0.9930
 757/1688 [============>.................] - ETA: 1s - loss: 0.0204 - accuracy: 0.9930
 787/1688 [============>.................] - ETA: 1s - loss: 0.0206 - accuracy: 0.9930
 817/1688 [=============>................] - ETA: 1s - loss: 0.0206 - accuracy: 0.9930
 847/1688 [==============>...............] - ETA: 1s - loss: 0.0203 - accuracy: 0.9931
 877/1688 [==============>...............] - ETA: 1s - loss: 0.0199 - accuracy: 0.9933
 907/1688 [===============>..............] - ETA: 1s - loss: 0.0197 - accuracy: 0.9934
 938/1688 [===============>..............] - ETA: 1s - loss: 0.0194 - accuracy: 0.9935
 969/1688 [================>.............] - ETA: 1s - loss: 0.0193 - accuracy: 0.9935
 998/1688 [================>.............] - ETA: 1s - loss: 0.0195 - accuracy: 0.9934
1029/1688 [=================>............] - ETA: 1s - loss: 0.0195 - accuracy: 0.9934
1059/1688 [=================>............] - ETA: 1s - loss: 0.0194 - accuracy: 0.9935
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0196 - accuracy: 0.9934
1119/1688 [==================>...........] - ETA: 0s - loss: 0.0195 - accuracy: 0.9934
1149/1688 [===================>..........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9933
1179/1688 [===================>..........] - ETA: 0s - loss: 0.0200 - accuracy: 0.9932
1209/1688 [====================>.........] - ETA: 0s - loss: 0.0203 - accuracy: 0.9930
1240/1688 [=====================>........] - ETA: 0s - loss: 0.0206 - accuracy: 0.9928
1271/1688 [=====================>........] - ETA: 0s - loss: 0.0210 - accuracy: 0.9928
1300/1688 [======================>.......] - ETA: 0s - loss: 0.0212 - accuracy: 0.9928
1331/1688 [======================>.......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9926
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9927
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9927
1422/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9927
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9926
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0220 - accuracy: 0.9926
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0220 - accuracy: 0.9925
1546/1688 [==========================>...] - ETA: 0s - loss: 0.0222 - accuracy: 0.9924
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0223 - accuracy: 0.9924
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0223 - accuracy: 0.9924
1637/1688 [============================>.] - ETA: 0s - loss: 0.0224 - accuracy: 0.9924
1667/1688 [============================>.] - ETA: 0s - loss: 0.0223 - accuracy: 0.9924
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0226 - accuracy: 0.9923 - val_loss: 0.0584 - val_accuracy: 0.9855
Epoch 8/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0214 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9961
  62/1688 [>.............................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9965
  93/1688 [>.............................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9953
 124/1688 [=>............................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9952
 154/1688 [=>............................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9951
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9954
 215/1688 [==>...........................] - ETA: 2s - loss: 0.0177 - accuracy: 0.9943
 245/1688 [===>..........................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9949
 275/1688 [===>..........................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9949
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9952
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9952
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9950
 397/1688 [======>.......................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9949
 427/1688 [======>.......................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9947
 457/1688 [=======>......................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9948
 489/1688 [=======>......................] - ETA: 1s - loss: 0.0157 - accuracy: 0.9945
 519/1688 [========>.....................] - ETA: 1s - loss: 0.0163 - accuracy: 0.9945
 549/1688 [========>.....................] - ETA: 1s - loss: 0.0165 - accuracy: 0.9944
 579/1688 [=========>....................] - ETA: 1s - loss: 0.0167 - accuracy: 0.9942
 610/1688 [=========>....................] - ETA: 1s - loss: 0.0167 - accuracy: 0.9943
 640/1688 [==========>...................] - ETA: 1s - loss: 0.0167 - accuracy: 0.9942
 670/1688 [==========>...................] - ETA: 1s - loss: 0.0167 - accuracy: 0.9942
 700/1688 [===========>..................] - ETA: 1s - loss: 0.0169 - accuracy: 0.9941
 730/1688 [===========>..................] - ETA: 1s - loss: 0.0170 - accuracy: 0.9940
 760/1688 [============>.................] - ETA: 1s - loss: 0.0172 - accuracy: 0.9940
 790/1688 [=============>................] - ETA: 1s - loss: 0.0175 - accuracy: 0.9939
 820/1688 [=============>................] - ETA: 1s - loss: 0.0173 - accuracy: 0.9940
 850/1688 [==============>...............] - ETA: 1s - loss: 0.0177 - accuracy: 0.9937
 880/1688 [==============>...............] - ETA: 1s - loss: 0.0179 - accuracy: 0.9937
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0179 - accuracy: 0.9938
 941/1688 [===============>..............] - ETA: 1s - loss: 0.0177 - accuracy: 0.9939
 970/1688 [================>.............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9939
1000/1688 [================>.............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9940
1031/1688 [=================>............] - ETA: 1s - loss: 0.0173 - accuracy: 0.9941
1061/1688 [=================>............] - ETA: 1s - loss: 0.0175 - accuracy: 0.9940
1092/1688 [==================>...........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9940
1123/1688 [==================>...........] - ETA: 0s - loss: 0.0176 - accuracy: 0.9941
1153/1688 [===================>..........] - ETA: 0s - loss: 0.0180 - accuracy: 0.9940
1183/1688 [====================>.........] - ETA: 0s - loss: 0.0181 - accuracy: 0.9939
1213/1688 [====================>.........] - ETA: 0s - loss: 0.0180 - accuracy: 0.9939
1244/1688 [=====================>........] - ETA: 0s - loss: 0.0183 - accuracy: 0.9938
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9938
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0187 - accuracy: 0.9937
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0186 - accuracy: 0.9937
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0186 - accuracy: 0.9937
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0185 - accuracy: 0.9938
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0189 - accuracy: 0.9935
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0191 - accuracy: 0.9935
1483/1688 [=========================>....] - ETA: 0s - loss: 0.0194 - accuracy: 0.9934
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0196 - accuracy: 0.9934
1543/1688 [==========================>...] - ETA: 0s - loss: 0.0196 - accuracy: 0.9934
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0196 - accuracy: 0.9934
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0194 - accuracy: 0.9935
1635/1688 [============================>.] - ETA: 0s - loss: 0.0195 - accuracy: 0.9934
1665/1688 [============================>.] - ETA: 0s - loss: 0.0196 - accuracy: 0.9934
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0196 - accuracy: 0.9934 - val_loss: 0.0562 - val_accuracy: 0.9868
Epoch 9/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0017 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0114 - accuracy: 0.9961
  62/1688 [>.............................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9940
  92/1688 [>.............................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9946
 123/1688 [=>............................] - ETA: 2s - loss: 0.0156 - accuracy: 0.9952
 154/1688 [=>............................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9951
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9954
 215/1688 [==>...........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9956
 245/1688 [===>..........................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9955
 275/1688 [===>..........................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9956
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9950
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9952
 364/1688 [=====>........................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9955
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9952
 424/1688 [======>.......................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9951
 454/1688 [=======>......................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9953
 484/1688 [=======>......................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9952
 514/1688 [========>.....................] - ETA: 1s - loss: 0.0149 - accuracy: 0.9950
 544/1688 [========>.....................] - ETA: 1s - loss: 0.0149 - accuracy: 0.9951
 573/1688 [=========>....................] - ETA: 1s - loss: 0.0150 - accuracy: 0.9950
 604/1688 [=========>....................] - ETA: 1s - loss: 0.0148 - accuracy: 0.9951
 634/1688 [==========>...................] - ETA: 1s - loss: 0.0150 - accuracy: 0.9950
 664/1688 [==========>...................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9952
 694/1688 [===========>..................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9953
 724/1688 [===========>..................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9953
 754/1688 [============>.................] - ETA: 1s - loss: 0.0146 - accuracy: 0.9952
 784/1688 [============>.................] - ETA: 1s - loss: 0.0147 - accuracy: 0.9952
 815/1688 [=============>................] - ETA: 1s - loss: 0.0151 - accuracy: 0.9952
 846/1688 [==============>...............] - ETA: 1s - loss: 0.0151 - accuracy: 0.9951
 877/1688 [==============>...............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9952
 907/1688 [===============>..............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9952
 940/1688 [===============>..............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9952
 971/1688 [================>.............] - ETA: 1s - loss: 0.0149 - accuracy: 0.9951
1001/1688 [================>.............] - ETA: 1s - loss: 0.0149 - accuracy: 0.9950
1031/1688 [=================>............] - ETA: 1s - loss: 0.0152 - accuracy: 0.9950
1061/1688 [=================>............] - ETA: 1s - loss: 0.0154 - accuracy: 0.9948
1091/1688 [==================>...........] - ETA: 1s - loss: 0.0155 - accuracy: 0.9947
1121/1688 [==================>...........] - ETA: 0s - loss: 0.0156 - accuracy: 0.9946
1153/1688 [===================>..........] - ETA: 0s - loss: 0.0156 - accuracy: 0.9946
1183/1688 [====================>.........] - ETA: 0s - loss: 0.0156 - accuracy: 0.9946
1213/1688 [====================>.........] - ETA: 0s - loss: 0.0156 - accuracy: 0.9945
1243/1688 [=====================>........] - ETA: 0s - loss: 0.0158 - accuracy: 0.9945
1273/1688 [=====================>........] - ETA: 0s - loss: 0.0158 - accuracy: 0.9946
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0161 - accuracy: 0.9944
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0160 - accuracy: 0.9945
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0162 - accuracy: 0.9944
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0166 - accuracy: 0.9944
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0168 - accuracy: 0.9943
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0169 - accuracy: 0.9943
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0168 - accuracy: 0.9943
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0166 - accuracy: 0.9944
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0167 - accuracy: 0.9943
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0166 - accuracy: 0.9943
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0167 - accuracy: 0.9943
1641/1688 [============================>.] - ETA: 0s - loss: 0.0166 - accuracy: 0.9943
1671/1688 [============================>.] - ETA: 0s - loss: 0.0168 - accuracy: 0.9942
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0169 - accuracy: 0.9942 - val_loss: 0.0643 - val_accuracy: 0.9865
Epoch 10/10

   1/1688 [..............................] - ETA: 2s - loss: 7.4646e-04 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0067 - accuracy: 0.9980    
  61/1688 [>.............................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9990
  92/1688 [>.............................] - ETA: 2s - loss: 0.0074 - accuracy: 0.9983
 123/1688 [=>............................] - ETA: 2s - loss: 0.0104 - accuracy: 0.9972
 153/1688 [=>............................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9951
 183/1688 [==>...........................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9954
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9953
 244/1688 [===>..........................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9954
 275/1688 [===>..........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9953
 306/1688 [====>.........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9948
 336/1688 [====>.........................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9949
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9951
 396/1688 [======>.......................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9950
 426/1688 [======>.......................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9952
 456/1688 [=======>......................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9955
 486/1688 [=======>......................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9956
 517/1688 [========>.....................] - ETA: 1s - loss: 0.0133 - accuracy: 0.9956
 547/1688 [========>.....................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9957
 576/1688 [=========>....................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9957
 607/1688 [=========>....................] - ETA: 1s - loss: 0.0130 - accuracy: 0.9958
 636/1688 [==========>...................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9956
 666/1688 [==========>...................] - ETA: 1s - loss: 0.0130 - accuracy: 0.9957
 695/1688 [===========>..................] - ETA: 1s - loss: 0.0128 - accuracy: 0.9957
 725/1688 [===========>..................] - ETA: 1s - loss: 0.0128 - accuracy: 0.9957
 756/1688 [============>.................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9955
 786/1688 [============>.................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9956
 816/1688 [=============>................] - ETA: 1s - loss: 0.0134 - accuracy: 0.9955
 847/1688 [==============>...............] - ETA: 1s - loss: 0.0132 - accuracy: 0.9956
 878/1688 [==============>...............] - ETA: 1s - loss: 0.0133 - accuracy: 0.9955
 908/1688 [===============>..............] - ETA: 1s - loss: 0.0135 - accuracy: 0.9956
 938/1688 [===============>..............] - ETA: 1s - loss: 0.0135 - accuracy: 0.9955
 968/1688 [================>.............] - ETA: 1s - loss: 0.0134 - accuracy: 0.9955
 998/1688 [================>.............] - ETA: 1s - loss: 0.0134 - accuracy: 0.9955
1028/1688 [=================>............] - ETA: 1s - loss: 0.0135 - accuracy: 0.9954
1058/1688 [=================>............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9955
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0135 - accuracy: 0.9954
1119/1688 [==================>...........] - ETA: 0s - loss: 0.0138 - accuracy: 0.9953
1150/1688 [===================>..........] - ETA: 0s - loss: 0.0138 - accuracy: 0.9954
1180/1688 [===================>..........] - ETA: 0s - loss: 0.0138 - accuracy: 0.9954
1211/1688 [====================>.........] - ETA: 0s - loss: 0.0141 - accuracy: 0.9951
1241/1688 [=====================>........] - ETA: 0s - loss: 0.0144 - accuracy: 0.9949
1271/1688 [=====================>........] - ETA: 0s - loss: 0.0145 - accuracy: 0.9950
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0146 - accuracy: 0.9948
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0146 - accuracy: 0.9948
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0147 - accuracy: 0.9948
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0146 - accuracy: 0.9948
1422/1688 [========================>.....] - ETA: 0s - loss: 0.0147 - accuracy: 0.9947
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0149 - accuracy: 0.9947
1483/1688 [=========================>....] - ETA: 0s - loss: 0.0149 - accuracy: 0.9946
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0148 - accuracy: 0.9947
1544/1688 [==========================>...] - ETA: 0s - loss: 0.0148 - accuracy: 0.9947
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0149 - accuracy: 0.9947
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0147 - accuracy: 0.9948
1635/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9948
1665/1688 [============================>.] - ETA: 0s - loss: 0.0147 - accuracy: 0.9948
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0147 - accuracy: 0.9948 - val_loss: 0.0587 - val_accuracy: 0.9870

score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])

Test accuracy: 0.9848999977111816

2. Quantize

2.1. 8-bit quantization

An Akida accelerator processes 8, 4, 2, or 1 bit integer activations and weights. Therefore, the floating point Keras model must be quantized in preparation to run on an Akida accelerator.

The QuantizeML quantize function can be used to quantize a Keras model for Akida. For this step in this example, an “8/8/8” quantization scheme will be applied to the floating point Keras model to produce 8-bit weights in the first layer, 8-bit weights in all other layers, and 8-bit activations.

The quantization process results in a Keras model with custom QuantizeML quantized layers substituted for the original Keras layers. All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function applies several transformations to the original model. For example, it folds the batch normalization layers into the corresponding neural layers. The new weights are computed according to this folding operation.

from quantizeml.models import quantize
from quantizeml.layers import QuantizationParams

qparams = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)
model_quantized = quantize(model_keras, qparams=qparams)
model_quantized.summary()

   1/1024 [..............................] - ETA: 2:26
  59/1024 [>.............................] - ETA: 0s  
 116/1024 [==>...........................] - ETA: 0s
 175/1024 [====>.........................] - ETA: 0s
 233/1024 [=====>........................] - ETA: 0s
 290/1024 [=======>......................] - ETA: 0s
 348/1024 [=========>....................] - ETA: 0s
 407/1024 [==========>...................] - ETA: 0s
 465/1024 [============>.................] - ETA: 0s
 524/1024 [==============>...............] - ETA: 0s
 582/1024 [================>.............] - ETA: 0s
 642/1024 [=================>............] - ETA: 0s
 701/1024 [===================>..........] - ETA: 0s
 759/1024 [=====================>........] - ETA: 0s
 817/1024 [======================>.......] - ETA: 0s
 875/1024 [========================>.....] - ETA: 0s
 933/1024 [==========================>...] - ETA: 0s
 992/1024 [============================>.] - ETA: 0s
1024/1024 [==============================] - 1s 864us/step
Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling_input (InputLayer  [(None, 28, 28, 1)]      0
 )

 rescaling (QuantizedRescali  (None, 28, 28, 1)        0
 ng)

 conv2d (QuantizedConv2D)    (None, 13, 13, 32)        320

 re_lu (QuantizedReLU)       (None, 13, 13, 32)        64

 depthwise_conv2d (Quantized  (None, 7, 7, 32)         384
 DepthwiseConv2D)

 conv2d_1 (QuantizedConv2D)  (None, 7, 7, 64)          2112

 re_lu_1 (QuantizedReLU)     (None, 7, 7, 64)          128

 flatten (QuantizedFlatten)  (None, 3136)              0

 dense (QuantizedDense)      (None, 10)                31370

 dequantizer (Dequantizer)   [(None, 10)]              0

=================================================================
Total params: 34,378
Trainable params: 34,122
Non-trainable params: 256
_________________________________________________________________

Note

Note that the number of parameters for the floating and quantized models differs, a consequence of the BatchNormalization folding and the additional parameters added for quantization. For further details, please refer to their respective summary.

Check the quantized model accuracy.

def compile_evaluate(model):
    """ Compiles and evaluates the model, then return accuracy score. """
    model.compile(metrics=['accuracy'])
    return model.evaluate(x_test, y_test, verbose=0)[1]


print('Test accuracy after 8-bit quantization:', compile_evaluate(model_quantized))

Test accuracy after 8-bit quantization: 0.9790999889373779

2.2. Effect of calibration

The previous call to quantize was made with random samples for calibration (default parameters). While the observed drop in accuracy is minimal, that is around 1%, it can be worse on more complex models. Therefore, it is advised to use a set of real samples from the training set for calibration during a call to quantize. Note that this remains a calibration step rather than a training step in that no output labels are required. Furthermore, any relevant data could be used for calibration. The recommended settings for calibration that are widely used to obtain the zoo performance are:

• 1024 samples

• a batch size of 100

• 2 epochs

model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)

 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 971us/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 964us/step

Check the accuracy for the quantized and calibrated model.

print('Test accuracy after calibration:', compile_evaluate(model_quantized))

Test accuracy after calibration: 0.9803000092506409

Calibrating with real samples on this model recovers the initial float accuracy.

2.3. 4-bit quantization

The accuracy of the 8/8/8 quantized model is equal to that of the Keras floating point model. In some cases, a smaller memory size for the model is required. This can be accomplished through quantization of the model to smaller bitwidths.

The model will now be quantized to 8/4/4, that is 8-bit weights in the first layer with 4-bit weights and activations in all other layers. Such a quantization scheme will usually introduce a performance drop.

qparams = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)
model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)

 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 979us/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 956us/step

Check the 4-bit quantized accuracy.

print('Test accuracy after 4-bit quantization:', compile_evaluate(model_quantized))

Test accuracy after 4-bit quantization: 0.9745000004768372

2.4. Model fine tuning (Quantization Aware Training)

When a model suffers from an accuracy drop after quantization, fine tuning or Quantization Aware Training (QAT) may recover some or all of the original performance.

Note that since this is a fine tuning step, both the number of epochs and learning rate are expected to be lower than during the initial float training.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-4),
    metrics=['accuracy'])

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)

Epoch 1/5

   1/1688 [..............................] - ETA: 1:03:51 - loss: 0.0442 - accuracy: 0.9688
  13/1688 [..............................] - ETA: 7s - loss: 0.0294 - accuracy: 0.9928     
  24/1688 [..............................] - ETA: 7s - loss: 0.0270 - accuracy: 0.9922
  36/1688 [..............................] - ETA: 7s - loss: 0.0306 - accuracy: 0.9896
  48/1688 [..............................] - ETA: 7s - loss: 0.0281 - accuracy: 0.9909
  60/1688 [>.............................] - ETA: 7s - loss: 0.0317 - accuracy: 0.9896
  72/1688 [>.............................] - ETA: 7s - loss: 0.0325 - accuracy: 0.9887
  84/1688 [>.............................] - ETA: 7s - loss: 0.0300 - accuracy: 0.9900
  96/1688 [>.............................] - ETA: 7s - loss: 0.0302 - accuracy: 0.9896
 108/1688 [>.............................] - ETA: 7s - loss: 0.0296 - accuracy: 0.9896
 120/1688 [=>............................] - ETA: 6s - loss: 0.0275 - accuracy: 0.9904
 132/1688 [=>............................] - ETA: 6s - loss: 0.0271 - accuracy: 0.9905
 144/1688 [=>............................] - ETA: 6s - loss: 0.0266 - accuracy: 0.9907
 156/1688 [=>............................] - ETA: 6s - loss: 0.0273 - accuracy: 0.9902
 167/1688 [=>............................] - ETA: 6s - loss: 0.0281 - accuracy: 0.9897
 179/1688 [==>...........................] - ETA: 6s - loss: 0.0269 - accuracy: 0.9900
 191/1688 [==>...........................] - ETA: 6s - loss: 0.0275 - accuracy: 0.9902
 203/1688 [==>...........................] - ETA: 6s - loss: 0.0263 - accuracy: 0.9908
 215/1688 [==>...........................] - ETA: 6s - loss: 0.0261 - accuracy: 0.9908
 227/1688 [===>..........................] - ETA: 6s - loss: 0.0253 - accuracy: 0.9913
 239/1688 [===>..........................] - ETA: 6s - loss: 0.0244 - accuracy: 0.9916
 251/1688 [===>..........................] - ETA: 6s - loss: 0.0244 - accuracy: 0.9918
 263/1688 [===>..........................] - ETA: 6s - loss: 0.0244 - accuracy: 0.9918
 275/1688 [===>..........................] - ETA: 6s - loss: 0.0238 - accuracy: 0.9919
 287/1688 [====>.........................] - ETA: 6s - loss: 0.0238 - accuracy: 0.9917
 299/1688 [====>.........................] - ETA: 6s - loss: 0.0242 - accuracy: 0.9914
 311/1688 [====>.........................] - ETA: 6s - loss: 0.0245 - accuracy: 0.9914
 323/1688 [====>.........................] - ETA: 6s - loss: 0.0245 - accuracy: 0.9914
 335/1688 [====>.........................] - ETA: 6s - loss: 0.0243 - accuracy: 0.9916
 347/1688 [=====>........................] - ETA: 5s - loss: 0.0238 - accuracy: 0.9918
 359/1688 [=====>........................] - ETA: 5s - loss: 0.0238 - accuracy: 0.9916
 370/1688 [=====>........................] - ETA: 5s - loss: 0.0233 - accuracy: 0.9919
 382/1688 [=====>........................] - ETA: 5s - loss: 0.0229 - accuracy: 0.9920
 394/1688 [======>.......................] - ETA: 5s - loss: 0.0231 - accuracy: 0.9920
 406/1688 [======>.......................] - ETA: 5s - loss: 0.0226 - accuracy: 0.9921
 418/1688 [======>.......................] - ETA: 5s - loss: 0.0225 - accuracy: 0.9921
 430/1688 [======>.......................] - ETA: 5s - loss: 0.0223 - accuracy: 0.9922
 442/1688 [======>.......................] - ETA: 5s - loss: 0.0221 - accuracy: 0.9923
 454/1688 [=======>......................] - ETA: 5s - loss: 0.0223 - accuracy: 0.9923
 466/1688 [=======>......................] - ETA: 5s - loss: 0.0221 - accuracy: 0.9925
 478/1688 [=======>......................] - ETA: 5s - loss: 0.0221 - accuracy: 0.9924
 490/1688 [=======>......................] - ETA: 5s - loss: 0.0219 - accuracy: 0.9925
 502/1688 [=======>......................] - ETA: 5s - loss: 0.0216 - accuracy: 0.9926
 514/1688 [========>.....................] - ETA: 5s - loss: 0.0218 - accuracy: 0.9924
 526/1688 [========>.....................] - ETA: 5s - loss: 0.0217 - accuracy: 0.9924
 538/1688 [========>.....................] - ETA: 5s - loss: 0.0216 - accuracy: 0.9924
 550/1688 [========>.....................] - ETA: 5s - loss: 0.0216 - accuracy: 0.9924
 562/1688 [========>.....................] - ETA: 4s - loss: 0.0214 - accuracy: 0.9925
 574/1688 [=========>....................] - ETA: 4s - loss: 0.0213 - accuracy: 0.9925
 586/1688 [=========>....................] - ETA: 4s - loss: 0.0212 - accuracy: 0.9927
 598/1688 [=========>....................] - ETA: 4s - loss: 0.0210 - accuracy: 0.9928
 610/1688 [=========>....................] - ETA: 4s - loss: 0.0210 - accuracy: 0.9928
 622/1688 [==========>...................] - ETA: 4s - loss: 0.0210 - accuracy: 0.9928
 634/1688 [==========>...................] - ETA: 4s - loss: 0.0208 - accuracy: 0.9929
 646/1688 [==========>...................] - ETA: 4s - loss: 0.0208 - accuracy: 0.9929
 658/1688 [==========>...................] - ETA: 4s - loss: 0.0208 - accuracy: 0.9929
 670/1688 [==========>...................] - ETA: 4s - loss: 0.0207 - accuracy: 0.9929
 682/1688 [===========>..................] - ETA: 4s - loss: 0.0206 - accuracy: 0.9930
 694/1688 [===========>..................] - ETA: 4s - loss: 0.0205 - accuracy: 0.9931
 706/1688 [===========>..................] - ETA: 4s - loss: 0.0205 - accuracy: 0.9931
 718/1688 [===========>..................] - ETA: 4s - loss: 0.0205 - accuracy: 0.9931
 730/1688 [===========>..................] - ETA: 4s - loss: 0.0205 - accuracy: 0.9931
 742/1688 [============>.................] - ETA: 4s - loss: 0.0204 - accuracy: 0.9932
 754/1688 [============>.................] - ETA: 4s - loss: 0.0207 - accuracy: 0.9930
 766/1688 [============>.................] - ETA: 4s - loss: 0.0206 - accuracy: 0.9931
 778/1688 [============>.................] - ETA: 4s - loss: 0.0204 - accuracy: 0.9932
 790/1688 [=============>................] - ETA: 3s - loss: 0.0203 - accuracy: 0.9932
 802/1688 [=============>................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9933
 814/1688 [=============>................] - ETA: 3s - loss: 0.0201 - accuracy: 0.9934
 826/1688 [=============>................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9933
 838/1688 [=============>................] - ETA: 3s - loss: 0.0201 - accuracy: 0.9934
 850/1688 [==============>...............] - ETA: 3s - loss: 0.0200 - accuracy: 0.9934
 862/1688 [==============>...............] - ETA: 3s - loss: 0.0201 - accuracy: 0.9934
 874/1688 [==============>...............] - ETA: 3s - loss: 0.0201 - accuracy: 0.9934
 886/1688 [==============>...............] - ETA: 3s - loss: 0.0200 - accuracy: 0.9934
 898/1688 [==============>...............] - ETA: 3s - loss: 0.0201 - accuracy: 0.9934
 910/1688 [===============>..............] - ETA: 3s - loss: 0.0201 - accuracy: 0.9934
 922/1688 [===============>..............] - ETA: 3s - loss: 0.0201 - accuracy: 0.9934
 933/1688 [===============>..............] - ETA: 3s - loss: 0.0203 - accuracy: 0.9933
 945/1688 [===============>..............] - ETA: 3s - loss: 0.0202 - accuracy: 0.9933
 957/1688 [================>.............] - ETA: 3s - loss: 0.0201 - accuracy: 0.9933
 969/1688 [================>.............] - ETA: 3s - loss: 0.0200 - accuracy: 0.9934
 981/1688 [================>.............] - ETA: 3s - loss: 0.0199 - accuracy: 0.9934
 993/1688 [================>.............] - ETA: 3s - loss: 0.0199 - accuracy: 0.9934
1005/1688 [================>.............] - ETA: 3s - loss: 0.0198 - accuracy: 0.9934
1017/1688 [=================>............] - ETA: 2s - loss: 0.0197 - accuracy: 0.9934
1029/1688 [=================>............] - ETA: 2s - loss: 0.0198 - accuracy: 0.9933
1041/1688 [=================>............] - ETA: 2s - loss: 0.0197 - accuracy: 0.9934
1053/1688 [=================>............] - ETA: 2s - loss: 0.0197 - accuracy: 0.9934
1065/1688 [=================>............] - ETA: 2s - loss: 0.0198 - accuracy: 0.9934
1077/1688 [==================>...........] - ETA: 2s - loss: 0.0197 - accuracy: 0.9935
1089/1688 [==================>...........] - ETA: 2s - loss: 0.0195 - accuracy: 0.9936
1101/1688 [==================>...........] - ETA: 2s - loss: 0.0195 - accuracy: 0.9936
1113/1688 [==================>...........] - ETA: 2s - loss: 0.0194 - accuracy: 0.9936
1125/1688 [==================>...........] - ETA: 2s - loss: 0.0194 - accuracy: 0.9936
1136/1688 [===================>..........] - ETA: 2s - loss: 0.0194 - accuracy: 0.9936
1148/1688 [===================>..........] - ETA: 2s - loss: 0.0194 - accuracy: 0.9936
1160/1688 [===================>..........] - ETA: 2s - loss: 0.0194 - accuracy: 0.9936
1172/1688 [===================>..........] - ETA: 2s - loss: 0.0195 - accuracy: 0.9935
1184/1688 [====================>.........] - ETA: 2s - loss: 0.0194 - accuracy: 0.9936
1196/1688 [====================>.........] - ETA: 2s - loss: 0.0193 - accuracy: 0.9936
1208/1688 [====================>.........] - ETA: 2s - loss: 0.0192 - accuracy: 0.9937
1220/1688 [====================>.........] - ETA: 2s - loss: 0.0192 - accuracy: 0.9936
1232/1688 [====================>.........] - ETA: 2s - loss: 0.0191 - accuracy: 0.9937
1244/1688 [=====================>........] - ETA: 1s - loss: 0.0190 - accuracy: 0.9937
1256/1688 [=====================>........] - ETA: 1s - loss: 0.0192 - accuracy: 0.9937
1268/1688 [=====================>........] - ETA: 1s - loss: 0.0192 - accuracy: 0.9937
1280/1688 [=====================>........] - ETA: 1s - loss: 0.0191 - accuracy: 0.9937
1292/1688 [=====================>........] - ETA: 1s - loss: 0.0191 - accuracy: 0.9937
1304/1688 [======================>.......] - ETA: 1s - loss: 0.0191 - accuracy: 0.9936
1316/1688 [======================>.......] - ETA: 1s - loss: 0.0192 - accuracy: 0.9936
1328/1688 [======================>.......] - ETA: 1s - loss: 0.0192 - accuracy: 0.9936
1340/1688 [======================>.......] - ETA: 1s - loss: 0.0192 - accuracy: 0.9936
1352/1688 [=======================>......] - ETA: 1s - loss: 0.0192 - accuracy: 0.9937
1364/1688 [=======================>......] - ETA: 1s - loss: 0.0192 - accuracy: 0.9937
1376/1688 [=======================>......] - ETA: 1s - loss: 0.0191 - accuracy: 0.9936
1388/1688 [=======================>......] - ETA: 1s - loss: 0.0193 - accuracy: 0.9936
1400/1688 [=======================>......] - ETA: 1s - loss: 0.0192 - accuracy: 0.9936
1412/1688 [========================>.....] - ETA: 1s - loss: 0.0192 - accuracy: 0.9936
1424/1688 [========================>.....] - ETA: 1s - loss: 0.0192 - accuracy: 0.9936
1436/1688 [========================>.....] - ETA: 1s - loss: 0.0191 - accuracy: 0.9937
1448/1688 [========================>.....] - ETA: 1s - loss: 0.0190 - accuracy: 0.9937
1460/1688 [========================>.....] - ETA: 1s - loss: 0.0190 - accuracy: 0.9937
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0191 - accuracy: 0.9937
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0191 - accuracy: 0.9937
1496/1688 [=========================>....] - ETA: 0s - loss: 0.0192 - accuracy: 0.9936
1508/1688 [=========================>....] - ETA: 0s - loss: 0.0193 - accuracy: 0.9936
1520/1688 [==========================>...] - ETA: 0s - loss: 0.0192 - accuracy: 0.9936
1532/1688 [==========================>...] - ETA: 0s - loss: 0.0192 - accuracy: 0.9936
1544/1688 [==========================>...] - ETA: 0s - loss: 0.0191 - accuracy: 0.9937
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0190 - accuracy: 0.9937
1568/1688 [==========================>...] - ETA: 0s - loss: 0.0189 - accuracy: 0.9937
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0189 - accuracy: 0.9937
1592/1688 [===========================>..] - ETA: 0s - loss: 0.0190 - accuracy: 0.9937
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0191 - accuracy: 0.9937
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0191 - accuracy: 0.9937
1627/1688 [===========================>..] - ETA: 0s - loss: 0.0191 - accuracy: 0.9937
1639/1688 [============================>.] - ETA: 0s - loss: 0.0190 - accuracy: 0.9938
1651/1688 [============================>.] - ETA: 0s - loss: 0.0190 - accuracy: 0.9938
1663/1688 [============================>.] - ETA: 0s - loss: 0.0189 - accuracy: 0.9938
1675/1688 [============================>.] - ETA: 0s - loss: 0.0189 - accuracy: 0.9939
1687/1688 [============================>.] - ETA: 0s - loss: 0.0189 - accuracy: 0.9939
1688/1688 [==============================] - 11s 5ms/step - loss: 0.0189 - accuracy: 0.9939 - val_loss: 0.0519 - val_accuracy: 0.9862
Epoch 2/5

   1/1688 [..............................] - ETA: 7s - loss: 0.0055 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0170 - accuracy: 0.9952
  25/1688 [..............................] - ETA: 7s - loss: 0.0172 - accuracy: 0.9925
  37/1688 [..............................] - ETA: 7s - loss: 0.0178 - accuracy: 0.9932
  49/1688 [..............................] - ETA: 7s - loss: 0.0152 - accuracy: 0.9949
  61/1688 [>.............................] - ETA: 7s - loss: 0.0155 - accuracy: 0.9944
  73/1688 [>.............................] - ETA: 7s - loss: 0.0152 - accuracy: 0.9949
  84/1688 [>.............................] - ETA: 7s - loss: 0.0151 - accuracy: 0.9952
  96/1688 [>.............................] - ETA: 7s - loss: 0.0144 - accuracy: 0.9958
 108/1688 [>.............................] - ETA: 6s - loss: 0.0144 - accuracy: 0.9957
 120/1688 [=>............................] - ETA: 6s - loss: 0.0145 - accuracy: 0.9956
 132/1688 [=>............................] - ETA: 6s - loss: 0.0147 - accuracy: 0.9955
 144/1688 [=>............................] - ETA: 6s - loss: 0.0147 - accuracy: 0.9957
 156/1688 [=>............................] - ETA: 6s - loss: 0.0144 - accuracy: 0.9958
 168/1688 [=>............................] - ETA: 6s - loss: 0.0140 - accuracy: 0.9961
 180/1688 [==>...........................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9958
 192/1688 [==>...........................] - ETA: 6s - loss: 0.0145 - accuracy: 0.9959
 204/1688 [==>...........................] - ETA: 6s - loss: 0.0147 - accuracy: 0.9959
 216/1688 [==>...........................] - ETA: 6s - loss: 0.0151 - accuracy: 0.9957
 228/1688 [===>..........................] - ETA: 6s - loss: 0.0151 - accuracy: 0.9955
 240/1688 [===>..........................] - ETA: 6s - loss: 0.0149 - accuracy: 0.9956
 252/1688 [===>..........................] - ETA: 6s - loss: 0.0146 - accuracy: 0.9958
 264/1688 [===>..........................] - ETA: 6s - loss: 0.0146 - accuracy: 0.9957
 276/1688 [===>..........................] - ETA: 6s - loss: 0.0144 - accuracy: 0.9959
 288/1688 [====>.........................] - ETA: 6s - loss: 0.0146 - accuracy: 0.9958
 300/1688 [====>.........................] - ETA: 6s - loss: 0.0145 - accuracy: 0.9958
 312/1688 [====>.........................] - ETA: 6s - loss: 0.0145 - accuracy: 0.9959
 324/1688 [====>.........................] - ETA: 6s - loss: 0.0147 - accuracy: 0.9958
 336/1688 [====>.........................] - ETA: 5s - loss: 0.0149 - accuracy: 0.9955
 348/1688 [=====>........................] - ETA: 5s - loss: 0.0150 - accuracy: 0.9954
 360/1688 [=====>........................] - ETA: 5s - loss: 0.0150 - accuracy: 0.9955
 372/1688 [=====>........................] - ETA: 5s - loss: 0.0148 - accuracy: 0.9955
 384/1688 [=====>........................] - ETA: 5s - loss: 0.0150 - accuracy: 0.9955
 396/1688 [======>.......................] - ETA: 5s - loss: 0.0149 - accuracy: 0.9955
 408/1688 [======>.......................] - ETA: 5s - loss: 0.0147 - accuracy: 0.9956
 420/1688 [======>.......................] - ETA: 5s - loss: 0.0146 - accuracy: 0.9956
 432/1688 [======>.......................] - ETA: 5s - loss: 0.0145 - accuracy: 0.9957
 444/1688 [======>.......................] - ETA: 5s - loss: 0.0144 - accuracy: 0.9958
 456/1688 [=======>......................] - ETA: 5s - loss: 0.0146 - accuracy: 0.9957
 468/1688 [=======>......................] - ETA: 5s - loss: 0.0147 - accuracy: 0.9957
 480/1688 [=======>......................] - ETA: 5s - loss: 0.0148 - accuracy: 0.9956
 492/1688 [=======>......................] - ETA: 5s - loss: 0.0150 - accuracy: 0.9956
 504/1688 [=======>......................] - ETA: 5s - loss: 0.0151 - accuracy: 0.9956
 516/1688 [========>.....................] - ETA: 5s - loss: 0.0149 - accuracy: 0.9956
 528/1688 [========>.....................] - ETA: 5s - loss: 0.0148 - accuracy: 0.9957
 540/1688 [========>.....................] - ETA: 5s - loss: 0.0149 - accuracy: 0.9957
 552/1688 [========>.....................] - ETA: 5s - loss: 0.0148 - accuracy: 0.9958
 564/1688 [=========>....................] - ETA: 4s - loss: 0.0148 - accuracy: 0.9958
 576/1688 [=========>....................] - ETA: 4s - loss: 0.0148 - accuracy: 0.9958
 588/1688 [=========>....................] - ETA: 4s - loss: 0.0148 - accuracy: 0.9958
 600/1688 [=========>....................] - ETA: 4s - loss: 0.0148 - accuracy: 0.9958
 612/1688 [=========>....................] - ETA: 4s - loss: 0.0147 - accuracy: 0.9958
 624/1688 [==========>...................] - ETA: 4s - loss: 0.0147 - accuracy: 0.9957
 636/1688 [==========>...................] - ETA: 4s - loss: 0.0146 - accuracy: 0.9958
 648/1688 [==========>...................] - ETA: 4s - loss: 0.0146 - accuracy: 0.9958
 660/1688 [==========>...................] - ETA: 4s - loss: 0.0145 - accuracy: 0.9959
 672/1688 [==========>...................] - ETA: 4s - loss: 0.0146 - accuracy: 0.9959
 684/1688 [===========>..................] - ETA: 4s - loss: 0.0146 - accuracy: 0.9958
 696/1688 [===========>..................] - ETA: 4s - loss: 0.0145 - accuracy: 0.9959
 708/1688 [===========>..................] - ETA: 4s - loss: 0.0146 - accuracy: 0.9959
 720/1688 [===========>..................] - ETA: 4s - loss: 0.0145 - accuracy: 0.9959
 732/1688 [============>.................] - ETA: 4s - loss: 0.0146 - accuracy: 0.9959
 744/1688 [============>.................] - ETA: 4s - loss: 0.0145 - accuracy: 0.9958
 756/1688 [============>.................] - ETA: 4s - loss: 0.0144 - accuracy: 0.9959
 768/1688 [============>.................] - ETA: 4s - loss: 0.0143 - accuracy: 0.9959
 780/1688 [============>.................] - ETA: 4s - loss: 0.0142 - accuracy: 0.9960
 792/1688 [=============>................] - ETA: 3s - loss: 0.0141 - accuracy: 0.9961
 804/1688 [=============>................] - ETA: 3s - loss: 0.0140 - accuracy: 0.9961
 816/1688 [=============>................] - ETA: 3s - loss: 0.0139 - accuracy: 0.9962
 828/1688 [=============>................] - ETA: 3s - loss: 0.0139 - accuracy: 0.9962
 840/1688 [=============>................] - ETA: 3s - loss: 0.0139 - accuracy: 0.9962
 852/1688 [==============>...............] - ETA: 3s - loss: 0.0137 - accuracy: 0.9962
 864/1688 [==============>...............] - ETA: 3s - loss: 0.0137 - accuracy: 0.9962
 876/1688 [==============>...............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9963
 888/1688 [==============>...............] - ETA: 3s - loss: 0.0137 - accuracy: 0.9963
 900/1688 [==============>...............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9962
 912/1688 [===============>..............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9962
 924/1688 [===============>..............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
 936/1688 [===============>..............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9962
 948/1688 [===============>..............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
 960/1688 [================>.............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9962
 972/1688 [================>.............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9962
 984/1688 [================>.............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
 996/1688 [================>.............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9962
1008/1688 [================>.............] - ETA: 3s - loss: 0.0140 - accuracy: 0.9961
1020/1688 [=================>............] - ETA: 2s - loss: 0.0139 - accuracy: 0.9961
1032/1688 [=================>............] - ETA: 2s - loss: 0.0138 - accuracy: 0.9962
1044/1688 [=================>............] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1056/1688 [=================>............] - ETA: 2s - loss: 0.0138 - accuracy: 0.9962
1068/1688 [=================>............] - ETA: 2s - loss: 0.0137 - accuracy: 0.9963
1080/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9963
1092/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9963
1104/1688 [==================>...........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9963
1116/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9963
1128/1688 [===================>..........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9963
1140/1688 [===================>..........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9963
1152/1688 [===================>..........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9961
1188/1688 [====================>.........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9961
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9961
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9962
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0139 - accuracy: 0.9961
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0139 - accuracy: 0.9961
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0138 - accuracy: 0.9961
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0139 - accuracy: 0.9961
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0139 - accuracy: 0.9961
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0139 - accuracy: 0.9961
1295/1688 [======================>.......] - ETA: 1s - loss: 0.0139 - accuracy: 0.9961
1307/1688 [======================>.......] - ETA: 1s - loss: 0.0140 - accuracy: 0.9961
1319/1688 [======================>.......] - ETA: 1s - loss: 0.0140 - accuracy: 0.9960
1331/1688 [======================>.......] - ETA: 1s - loss: 0.0141 - accuracy: 0.9960
1343/1688 [======================>.......] - ETA: 1s - loss: 0.0143 - accuracy: 0.9959
1355/1688 [=======================>......] - ETA: 1s - loss: 0.0142 - accuracy: 0.9959
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0141 - accuracy: 0.9960
1379/1688 [=======================>......] - ETA: 1s - loss: 0.0141 - accuracy: 0.9960
1391/1688 [=======================>......] - ETA: 1s - loss: 0.0141 - accuracy: 0.9960
1403/1688 [=======================>......] - ETA: 1s - loss: 0.0141 - accuracy: 0.9959
1415/1688 [========================>.....] - ETA: 1s - loss: 0.0142 - accuracy: 0.9960
1427/1688 [========================>.....] - ETA: 1s - loss: 0.0142 - accuracy: 0.9959
1439/1688 [========================>.....] - ETA: 1s - loss: 0.0141 - accuracy: 0.9959
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0142 - accuracy: 0.9959
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0143 - accuracy: 0.9958
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0143 - accuracy: 0.9959
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0141 - accuracy: 0.9960
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0142 - accuracy: 0.9960
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0141 - accuracy: 0.9959
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0141 - accuracy: 0.9959
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0141 - accuracy: 0.9959
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0141 - accuracy: 0.9959
1643/1688 [============================>.] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1655/1688 [============================>.] - ETA: 0s - loss: 0.0143 - accuracy: 0.9959
1667/1688 [============================>.] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1679/1688 [============================>.] - ETA: 0s - loss: 0.0142 - accuracy: 0.9959
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0143 - accuracy: 0.9959 - val_loss: 0.0508 - val_accuracy: 0.9878
Epoch 3/5

   1/1688 [..............................] - ETA: 7s - loss: 6.7207e-04 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0115 - accuracy: 0.9976    
  25/1688 [..............................] - ETA: 7s - loss: 0.0126 - accuracy: 0.9975
  37/1688 [..............................] - ETA: 7s - loss: 0.0146 - accuracy: 0.9958
  49/1688 [..............................] - ETA: 7s - loss: 0.0131 - accuracy: 0.9962
  61/1688 [>.............................] - ETA: 7s - loss: 0.0172 - accuracy: 0.9939
  73/1688 [>.............................] - ETA: 7s - loss: 0.0165 - accuracy: 0.9940
  85/1688 [>.............................] - ETA: 7s - loss: 0.0157 - accuracy: 0.9945
  97/1688 [>.............................] - ETA: 7s - loss: 0.0149 - accuracy: 0.9952
 109/1688 [>.............................] - ETA: 6s - loss: 0.0145 - accuracy: 0.9954
 121/1688 [=>............................] - ETA: 6s - loss: 0.0141 - accuracy: 0.9956
 133/1688 [=>............................] - ETA: 6s - loss: 0.0139 - accuracy: 0.9958
 145/1688 [=>............................] - ETA: 6s - loss: 0.0140 - accuracy: 0.9957
 157/1688 [=>............................] - ETA: 6s - loss: 0.0136 - accuracy: 0.9960
 169/1688 [==>...........................] - ETA: 6s - loss: 0.0131 - accuracy: 0.9961
 181/1688 [==>...........................] - ETA: 6s - loss: 0.0134 - accuracy: 0.9960
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0133 - accuracy: 0.9961
 205/1688 [==>...........................] - ETA: 6s - loss: 0.0139 - accuracy: 0.9962
 217/1688 [==>...........................] - ETA: 6s - loss: 0.0139 - accuracy: 0.9961
 229/1688 [===>..........................] - ETA: 6s - loss: 0.0140 - accuracy: 0.9958
 241/1688 [===>..........................] - ETA: 6s - loss: 0.0136 - accuracy: 0.9960
 253/1688 [===>..........................] - ETA: 6s - loss: 0.0136 - accuracy: 0.9959
 265/1688 [===>..........................] - ETA: 6s - loss: 0.0135 - accuracy: 0.9960
 277/1688 [===>..........................] - ETA: 6s - loss: 0.0134 - accuracy: 0.9961
 289/1688 [====>.........................] - ETA: 6s - loss: 0.0133 - accuracy: 0.9962
 301/1688 [====>.........................] - ETA: 6s - loss: 0.0135 - accuracy: 0.9962
 313/1688 [====>.........................] - ETA: 6s - loss: 0.0133 - accuracy: 0.9963
 325/1688 [====>.........................] - ETA: 6s - loss: 0.0136 - accuracy: 0.9962
 337/1688 [====>.........................] - ETA: 5s - loss: 0.0137 - accuracy: 0.9960
 349/1688 [=====>........................] - ETA: 5s - loss: 0.0137 - accuracy: 0.9961
 361/1688 [=====>........................] - ETA: 5s - loss: 0.0136 - accuracy: 0.9961
 373/1688 [=====>........................] - ETA: 5s - loss: 0.0137 - accuracy: 0.9959
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0138 - accuracy: 0.9959
 397/1688 [======>.......................] - ETA: 5s - loss: 0.0137 - accuracy: 0.9961
 409/1688 [======>.......................] - ETA: 5s - loss: 0.0136 - accuracy: 0.9960
 421/1688 [======>.......................] - ETA: 5s - loss: 0.0138 - accuracy: 0.9960
 433/1688 [======>.......................] - ETA: 5s - loss: 0.0140 - accuracy: 0.9960
 445/1688 [======>.......................] - ETA: 5s - loss: 0.0139 - accuracy: 0.9961
 457/1688 [=======>......................] - ETA: 5s - loss: 0.0138 - accuracy: 0.9961
 469/1688 [=======>......................] - ETA: 5s - loss: 0.0137 - accuracy: 0.9962
 481/1688 [=======>......................] - ETA: 5s - loss: 0.0136 - accuracy: 0.9962
 493/1688 [=======>......................] - ETA: 5s - loss: 0.0134 - accuracy: 0.9963
 505/1688 [=======>......................] - ETA: 5s - loss: 0.0134 - accuracy: 0.9963
 517/1688 [========>.....................] - ETA: 5s - loss: 0.0137 - accuracy: 0.9962
 529/1688 [========>.....................] - ETA: 5s - loss: 0.0135 - accuracy: 0.9963
 541/1688 [========>.....................] - ETA: 5s - loss: 0.0137 - accuracy: 0.9962
 553/1688 [========>.....................] - ETA: 5s - loss: 0.0140 - accuracy: 0.9961
 565/1688 [=========>....................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9962
 577/1688 [=========>....................] - ETA: 4s - loss: 0.0137 - accuracy: 0.9963
 589/1688 [=========>....................] - ETA: 4s - loss: 0.0137 - accuracy: 0.9963
 601/1688 [=========>....................] - ETA: 4s - loss: 0.0137 - accuracy: 0.9962
 613/1688 [=========>....................] - ETA: 4s - loss: 0.0138 - accuracy: 0.9962
 625/1688 [==========>...................] - ETA: 4s - loss: 0.0138 - accuracy: 0.9962
 637/1688 [==========>...................] - ETA: 4s - loss: 0.0140 - accuracy: 0.9961
 649/1688 [==========>...................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9961
 661/1688 [==========>...................] - ETA: 4s - loss: 0.0141 - accuracy: 0.9961
 673/1688 [==========>...................] - ETA: 4s - loss: 0.0141 - accuracy: 0.9961
 685/1688 [===========>..................] - ETA: 4s - loss: 0.0141 - accuracy: 0.9961
 697/1688 [===========>..................] - ETA: 4s - loss: 0.0140 - accuracy: 0.9961
 709/1688 [===========>..................] - ETA: 4s - loss: 0.0140 - accuracy: 0.9961
 721/1688 [===========>..................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9962
 733/1688 [============>.................] - ETA: 4s - loss: 0.0140 - accuracy: 0.9961
 745/1688 [============>.................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9961
 757/1688 [============>.................] - ETA: 4s - loss: 0.0140 - accuracy: 0.9961
 769/1688 [============>.................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9961
 781/1688 [============>.................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9960
 793/1688 [=============>................] - ETA: 3s - loss: 0.0140 - accuracy: 0.9960
 805/1688 [=============>................] - ETA: 3s - loss: 0.0138 - accuracy: 0.9960
 817/1688 [=============>................] - ETA: 3s - loss: 0.0137 - accuracy: 0.9961
 829/1688 [=============>................] - ETA: 3s - loss: 0.0137 - accuracy: 0.9960
 841/1688 [=============>................] - ETA: 3s - loss: 0.0136 - accuracy: 0.9961
 853/1688 [==============>...............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9961
 865/1688 [==============>...............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9961
 877/1688 [==============>...............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9962
 889/1688 [==============>...............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9962
 901/1688 [===============>..............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9962
 913/1688 [===============>..............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9962
 925/1688 [===============>..............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9961
 937/1688 [===============>..............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9961
 949/1688 [===============>..............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9961
 961/1688 [================>.............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9960
 973/1688 [================>.............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9961
 985/1688 [================>.............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9961
 997/1688 [================>.............] - ETA: 3s - loss: 0.0133 - accuracy: 0.9961
1009/1688 [================>.............] - ETA: 2s - loss: 0.0133 - accuracy: 0.9961
1021/1688 [=================>............] - ETA: 2s - loss: 0.0134 - accuracy: 0.9961
1033/1688 [=================>............] - ETA: 2s - loss: 0.0133 - accuracy: 0.9962
1045/1688 [=================>............] - ETA: 2s - loss: 0.0133 - accuracy: 0.9962
1057/1688 [=================>............] - ETA: 2s - loss: 0.0133 - accuracy: 0.9962
1069/1688 [=================>............] - ETA: 2s - loss: 0.0132 - accuracy: 0.9962
1081/1688 [==================>...........] - ETA: 2s - loss: 0.0134 - accuracy: 0.9962
1093/1688 [==================>...........] - ETA: 2s - loss: 0.0134 - accuracy: 0.9962
1105/1688 [==================>...........] - ETA: 2s - loss: 0.0134 - accuracy: 0.9961
1117/1688 [==================>...........] - ETA: 2s - loss: 0.0135 - accuracy: 0.9961
1129/1688 [===================>..........] - ETA: 2s - loss: 0.0134 - accuracy: 0.9961
1141/1688 [===================>..........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9962
1153/1688 [===================>..........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9962
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9962
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9962
1188/1688 [====================>.........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9962
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9961
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0134 - accuracy: 0.9961
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0134 - accuracy: 0.9961
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0135 - accuracy: 0.9961
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0135 - accuracy: 0.9961
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0135 - accuracy: 0.9961
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0134 - accuracy: 0.9961
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1308/1688 [======================>.......] - ETA: 1s - loss: 0.0132 - accuracy: 0.9962
1320/1688 [======================>.......] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0132 - accuracy: 0.9963
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0131 - accuracy: 0.9963
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0130 - accuracy: 0.9963
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0130 - accuracy: 0.9963
1380/1688 [=======================>......] - ETA: 1s - loss: 0.0130 - accuracy: 0.9964
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0129 - accuracy: 0.9964
1404/1688 [=======================>......] - ETA: 1s - loss: 0.0130 - accuracy: 0.9963
1416/1688 [========================>.....] - ETA: 1s - loss: 0.0130 - accuracy: 0.9963
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0129 - accuracy: 0.9963
1440/1688 [========================>.....] - ETA: 1s - loss: 0.0129 - accuracy: 0.9964
1452/1688 [========================>.....] - ETA: 1s - loss: 0.0129 - accuracy: 0.9964
1464/1688 [=========================>....] - ETA: 0s - loss: 0.0128 - accuracy: 0.9964
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0128 - accuracy: 0.9964
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0128 - accuracy: 0.9965
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0128 - accuracy: 0.9965
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0128 - accuracy: 0.9964
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0128 - accuracy: 0.9965
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0127 - accuracy: 0.9965
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0128 - accuracy: 0.9964
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0128 - accuracy: 0.9965
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0127 - accuracy: 0.9965
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0127 - accuracy: 0.9964
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0129 - accuracy: 0.9964
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0129 - accuracy: 0.9964
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0129 - accuracy: 0.9964
1632/1688 [============================>.] - ETA: 0s - loss: 0.0129 - accuracy: 0.9964
1644/1688 [============================>.] - ETA: 0s - loss: 0.0128 - accuracy: 0.9964
1656/1688 [============================>.] - ETA: 0s - loss: 0.0129 - accuracy: 0.9964
1668/1688 [============================>.] - ETA: 0s - loss: 0.0129 - accuracy: 0.9964
1680/1688 [============================>.] - ETA: 0s - loss: 0.0128 - accuracy: 0.9964
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0129 - accuracy: 0.9964 - val_loss: 0.0499 - val_accuracy: 0.9880
Epoch 4/5

   1/1688 [..............................] - ETA: 7s - loss: 0.0013 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0159 - accuracy: 0.9904
  25/1688 [..............................] - ETA: 7s - loss: 0.0145 - accuracy: 0.9925
  37/1688 [..............................] - ETA: 7s - loss: 0.0117 - accuracy: 0.9949
  49/1688 [..............................] - ETA: 7s - loss: 0.0096 - accuracy: 0.9962
  61/1688 [>.............................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9959
  73/1688 [>.............................] - ETA: 7s - loss: 0.0095 - accuracy: 0.9966
  85/1688 [>.............................] - ETA: 7s - loss: 0.0105 - accuracy: 0.9960
  97/1688 [>.............................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9965
 109/1688 [>.............................] - ETA: 6s - loss: 0.0100 - accuracy: 0.9966
 121/1688 [=>............................] - ETA: 6s - loss: 0.0103 - accuracy: 0.9966
 133/1688 [=>............................] - ETA: 6s - loss: 0.0102 - accuracy: 0.9969
 145/1688 [=>............................] - ETA: 6s - loss: 0.0103 - accuracy: 0.9968
 157/1688 [=>............................] - ETA: 6s - loss: 0.0103 - accuracy: 0.9968
 169/1688 [==>...........................] - ETA: 6s - loss: 0.0116 - accuracy: 0.9965
 181/1688 [==>...........................] - ETA: 6s - loss: 0.0115 - accuracy: 0.9965
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0113 - accuracy: 0.9966
 205/1688 [==>...........................] - ETA: 6s - loss: 0.0113 - accuracy: 0.9965
 217/1688 [==>...........................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9963
 229/1688 [===>..........................] - ETA: 6s - loss: 0.0124 - accuracy: 0.9962
 241/1688 [===>..........................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9964
 253/1688 [===>..........................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9963
 265/1688 [===>..........................] - ETA: 6s - loss: 0.0119 - accuracy: 0.9965
 277/1688 [===>..........................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9966
 289/1688 [====>.........................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9966
 301/1688 [====>.........................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9966
 313/1688 [====>.........................] - ETA: 6s - loss: 0.0115 - accuracy: 0.9966
 325/1688 [====>.........................] - ETA: 6s - loss: 0.0113 - accuracy: 0.9967
 337/1688 [====>.........................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9968
 349/1688 [=====>........................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9969
 361/1688 [=====>........................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9969
 373/1688 [=====>........................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9968
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9967
 397/1688 [======>.......................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9967
 409/1688 [======>.......................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9968
 421/1688 [======>.......................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9967
 433/1688 [======>.......................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9968
 445/1688 [======>.......................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9968
 457/1688 [=======>......................] - ETA: 5s - loss: 0.0110 - accuracy: 0.9968
 469/1688 [=======>......................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9967
 481/1688 [=======>......................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9967
 493/1688 [=======>......................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9966
 505/1688 [=======>......................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9967
 517/1688 [========>.....................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9966
 529/1688 [========>.....................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9966
 541/1688 [========>.....................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9966
 553/1688 [========>.....................] - ETA: 5s - loss: 0.0116 - accuracy: 0.9966
 565/1688 [=========>....................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9965
 577/1688 [=========>....................] - ETA: 4s - loss: 0.0115 - accuracy: 0.9966
 589/1688 [=========>....................] - ETA: 4s - loss: 0.0116 - accuracy: 0.9966
 601/1688 [=========>....................] - ETA: 4s - loss: 0.0116 - accuracy: 0.9966
 613/1688 [=========>....................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9964
 625/1688 [==========>...................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9964
 637/1688 [==========>...................] - ETA: 4s - loss: 0.0120 - accuracy: 0.9964
 649/1688 [==========>...................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9964
 661/1688 [==========>...................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9964
 673/1688 [==========>...................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9965
 685/1688 [===========>..................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9965
 697/1688 [===========>..................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9965
 709/1688 [===========>..................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9965
 721/1688 [===========>..................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9965
 733/1688 [============>.................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9965
 745/1688 [============>.................] - ETA: 4s - loss: 0.0116 - accuracy: 0.9966
 757/1688 [============>.................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9966
 769/1688 [============>.................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9966
 781/1688 [============>.................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9966
 792/1688 [=============>................] - ETA: 3s - loss: 0.0117 - accuracy: 0.9966
 804/1688 [=============>................] - ETA: 3s - loss: 0.0116 - accuracy: 0.9966
 816/1688 [=============>................] - ETA: 3s - loss: 0.0116 - accuracy: 0.9966
 828/1688 [=============>................] - ETA: 3s - loss: 0.0117 - accuracy: 0.9966
 840/1688 [=============>................] - ETA: 3s - loss: 0.0116 - accuracy: 0.9966
 852/1688 [==============>...............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9966
 864/1688 [==============>...............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9967
 876/1688 [==============>...............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9967
 888/1688 [==============>...............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9967
 900/1688 [==============>...............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9967
 912/1688 [===============>..............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9967
 924/1688 [===============>..............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9967
 936/1688 [===============>..............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9967
 948/1688 [===============>..............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9967
 960/1688 [================>.............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9967
 972/1688 [================>.............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9967
 984/1688 [================>.............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9967
 996/1688 [================>.............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9967
1008/1688 [================>.............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9967
1020/1688 [=================>............] - ETA: 2s - loss: 0.0114 - accuracy: 0.9967
1032/1688 [=================>............] - ETA: 2s - loss: 0.0116 - accuracy: 0.9967
1044/1688 [=================>............] - ETA: 2s - loss: 0.0116 - accuracy: 0.9967
1056/1688 [=================>............] - ETA: 2s - loss: 0.0116 - accuracy: 0.9967
1068/1688 [=================>............] - ETA: 2s - loss: 0.0115 - accuracy: 0.9968
1080/1688 [==================>...........] - ETA: 2s - loss: 0.0115 - accuracy: 0.9968
1092/1688 [==================>...........] - ETA: 2s - loss: 0.0116 - accuracy: 0.9967
1104/1688 [==================>...........] - ETA: 2s - loss: 0.0116 - accuracy: 0.9967
1116/1688 [==================>...........] - ETA: 2s - loss: 0.0116 - accuracy: 0.9967
1128/1688 [===================>..........] - ETA: 2s - loss: 0.0116 - accuracy: 0.9967
1140/1688 [===================>..........] - ETA: 2s - loss: 0.0119 - accuracy: 0.9967
1152/1688 [===================>..........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9966
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9966
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9965
1188/1688 [====================>.........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9965
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0121 - accuracy: 0.9965
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0121 - accuracy: 0.9965
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0122 - accuracy: 0.9965
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0121 - accuracy: 0.9965
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0121 - accuracy: 0.9965
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0120 - accuracy: 0.9965
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0120 - accuracy: 0.9966
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0119 - accuracy: 0.9966
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9966
1308/1688 [======================>.......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9966
1320/1688 [======================>.......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9966
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9966
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9966
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9966
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0118 - accuracy: 0.9966
1380/1688 [=======================>......] - ETA: 1s - loss: 0.0118 - accuracy: 0.9966
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0118 - accuracy: 0.9967
1404/1688 [=======================>......] - ETA: 1s - loss: 0.0118 - accuracy: 0.9966
1416/1688 [========================>.....] - ETA: 1s - loss: 0.0118 - accuracy: 0.9966
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0117 - accuracy: 0.9967
1440/1688 [========================>.....] - ETA: 1s - loss: 0.0118 - accuracy: 0.9966
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0119 - accuracy: 0.9966
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9966
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9966
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9966
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9966
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0118 - accuracy: 0.9966
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9966
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9966
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9966
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9966
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0118 - accuracy: 0.9966
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0118 - accuracy: 0.9966
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0118 - accuracy: 0.9966
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0118 - accuracy: 0.9967
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0118 - accuracy: 0.9967
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0118 - accuracy: 0.9967
1643/1688 [============================>.] - ETA: 0s - loss: 0.0118 - accuracy: 0.9967
1655/1688 [============================>.] - ETA: 0s - loss: 0.0117 - accuracy: 0.9967
1667/1688 [============================>.] - ETA: 0s - loss: 0.0118 - accuracy: 0.9967
1679/1688 [============================>.] - ETA: 0s - loss: 0.0118 - accuracy: 0.9967
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0119 - accuracy: 0.9966 - val_loss: 0.0500 - val_accuracy: 0.9872
Epoch 5/5

   1/1688 [..............................] - ETA: 7s - loss: 0.0076 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0148 - accuracy: 0.9952
  25/1688 [..............................] - ETA: 7s - loss: 0.0149 - accuracy: 0.9950
  37/1688 [..............................] - ETA: 7s - loss: 0.0181 - accuracy: 0.9941
  49/1688 [..............................] - ETA: 7s - loss: 0.0183 - accuracy: 0.9936
  61/1688 [>.............................] - ETA: 7s - loss: 0.0191 - accuracy: 0.9923
  73/1688 [>.............................] - ETA: 7s - loss: 0.0172 - accuracy: 0.9932
  85/1688 [>.............................] - ETA: 7s - loss: 0.0162 - accuracy: 0.9941
  97/1688 [>.............................] - ETA: 7s - loss: 0.0152 - accuracy: 0.9948
 109/1688 [>.............................] - ETA: 6s - loss: 0.0149 - accuracy: 0.9948
 121/1688 [=>............................] - ETA: 6s - loss: 0.0158 - accuracy: 0.9948
 133/1688 [=>............................] - ETA: 6s - loss: 0.0151 - accuracy: 0.9953
 145/1688 [=>............................] - ETA: 6s - loss: 0.0148 - accuracy: 0.9953
 157/1688 [=>............................] - ETA: 6s - loss: 0.0142 - accuracy: 0.9956
 169/1688 [==>...........................] - ETA: 6s - loss: 0.0137 - accuracy: 0.9959
 181/1688 [==>...........................] - ETA: 6s - loss: 0.0135 - accuracy: 0.9959
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0137 - accuracy: 0.9958
 205/1688 [==>...........................] - ETA: 6s - loss: 0.0132 - accuracy: 0.9960
 217/1688 [==>...........................] - ETA: 6s - loss: 0.0135 - accuracy: 0.9960
 229/1688 [===>..........................] - ETA: 6s - loss: 0.0130 - accuracy: 0.9962
 241/1688 [===>..........................] - ETA: 6s - loss: 0.0133 - accuracy: 0.9962
 253/1688 [===>..........................] - ETA: 6s - loss: 0.0128 - accuracy: 0.9964
 265/1688 [===>..........................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9966
 277/1688 [===>..........................] - ETA: 6s - loss: 0.0124 - accuracy: 0.9967
 289/1688 [====>.........................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9968
 301/1688 [====>.........................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9968
 313/1688 [====>.........................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9969
 325/1688 [====>.........................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9969
 337/1688 [====>.........................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9968
 349/1688 [=====>........................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9967
 361/1688 [=====>........................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9968
 373/1688 [=====>........................] - ETA: 5s - loss: 0.0118 - accuracy: 0.9968
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0121 - accuracy: 0.9968
 397/1688 [======>.......................] - ETA: 5s - loss: 0.0124 - accuracy: 0.9966
 409/1688 [======>.......................] - ETA: 5s - loss: 0.0124 - accuracy: 0.9966
 421/1688 [======>.......................] - ETA: 5s - loss: 0.0124 - accuracy: 0.9965
 433/1688 [======>.......................] - ETA: 5s - loss: 0.0124 - accuracy: 0.9965
 445/1688 [======>.......................] - ETA: 5s - loss: 0.0124 - accuracy: 0.9966
 457/1688 [=======>......................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9966
 469/1688 [=======>......................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9966
 481/1688 [=======>......................] - ETA: 5s - loss: 0.0121 - accuracy: 0.9966
 493/1688 [=======>......................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9966
 505/1688 [=======>......................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9967
 517/1688 [========>.....................] - ETA: 5s - loss: 0.0121 - accuracy: 0.9965
 529/1688 [========>.....................] - ETA: 5s - loss: 0.0120 - accuracy: 0.9966
 541/1688 [========>.....................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9966
 553/1688 [========>.....................] - ETA: 5s - loss: 0.0117 - accuracy: 0.9967
 565/1688 [=========>....................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9967
 577/1688 [=========>....................] - ETA: 4s - loss: 0.0116 - accuracy: 0.9967
 589/1688 [=========>....................] - ETA: 4s - loss: 0.0116 - accuracy: 0.9967
 601/1688 [=========>....................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9965
 613/1688 [=========>....................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9965
 625/1688 [==========>...................] - ETA: 4s - loss: 0.0116 - accuracy: 0.9966
 637/1688 [==========>...................] - ETA: 4s - loss: 0.0115 - accuracy: 0.9967
 649/1688 [==========>...................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9966
 661/1688 [==========>...................] - ETA: 4s - loss: 0.0115 - accuracy: 0.9966
 672/1688 [==========>...................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9967
 684/1688 [===========>..................] - ETA: 4s - loss: 0.0113 - accuracy: 0.9967
 696/1688 [===========>..................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9966
 708/1688 [===========>..................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9966
 720/1688 [===========>..................] - ETA: 4s - loss: 0.0115 - accuracy: 0.9966
 732/1688 [============>.................] - ETA: 4s - loss: 0.0115 - accuracy: 0.9966
 744/1688 [============>.................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9966
 756/1688 [============>.................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9967
 768/1688 [============>.................] - ETA: 4s - loss: 0.0113 - accuracy: 0.9967
 780/1688 [============>.................] - ETA: 4s - loss: 0.0112 - accuracy: 0.9967
 792/1688 [=============>................] - ETA: 3s - loss: 0.0113 - accuracy: 0.9967
 804/1688 [=============>................] - ETA: 3s - loss: 0.0115 - accuracy: 0.9967
 816/1688 [=============>................] - ETA: 3s - loss: 0.0117 - accuracy: 0.9966
 828/1688 [=============>................] - ETA: 3s - loss: 0.0117 - accuracy: 0.9966
 840/1688 [=============>................] - ETA: 3s - loss: 0.0117 - accuracy: 0.9966
 852/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9966
 864/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9967
 876/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9967
 888/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9967
 900/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9967
 912/1688 [===============>..............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9967
 924/1688 [===============>..............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9967
 936/1688 [===============>..............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9967
 948/1688 [===============>..............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9967
 960/1688 [================>.............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9968
 972/1688 [================>.............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9968
 984/1688 [================>.............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9968
 996/1688 [================>.............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9969
1008/1688 [================>.............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9969
1020/1688 [=================>............] - ETA: 2s - loss: 0.0114 - accuracy: 0.9969
1032/1688 [=================>............] - ETA: 2s - loss: 0.0114 - accuracy: 0.9969
1044/1688 [=================>............] - ETA: 2s - loss: 0.0114 - accuracy: 0.9969
1056/1688 [=================>............] - ETA: 2s - loss: 0.0113 - accuracy: 0.9969
1068/1688 [=================>............] - ETA: 2s - loss: 0.0114 - accuracy: 0.9969
1080/1688 [==================>...........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9969
1092/1688 [==================>...........] - ETA: 2s - loss: 0.0115 - accuracy: 0.9969
1104/1688 [==================>...........] - ETA: 2s - loss: 0.0115 - accuracy: 0.9968
1116/1688 [==================>...........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9968
1128/1688 [===================>..........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9968
1140/1688 [===================>..........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9968
1152/1688 [===================>..........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9968
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9968
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0113 - accuracy: 0.9968
1188/1688 [====================>.........] - ETA: 2s - loss: 0.0112 - accuracy: 0.9969
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0113 - accuracy: 0.9968
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0113 - accuracy: 0.9968
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0113 - accuracy: 0.9968
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0113 - accuracy: 0.9968
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0112 - accuracy: 0.9968
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0113 - accuracy: 0.9968
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0113 - accuracy: 0.9968
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0113 - accuracy: 0.9968
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9968
1308/1688 [======================>.......] - ETA: 1s - loss: 0.0113 - accuracy: 0.9968
1320/1688 [======================>.......] - ETA: 1s - loss: 0.0113 - accuracy: 0.9968
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0113 - accuracy: 0.9968
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0113 - accuracy: 0.9968
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9968
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0113 - accuracy: 0.9968
1379/1688 [=======================>......] - ETA: 1s - loss: 0.0114 - accuracy: 0.9968
1391/1688 [=======================>......] - ETA: 1s - loss: 0.0114 - accuracy: 0.9968
1403/1688 [=======================>......] - ETA: 1s - loss: 0.0114 - accuracy: 0.9968
1415/1688 [========================>.....] - ETA: 1s - loss: 0.0114 - accuracy: 0.9968
1427/1688 [========================>.....] - ETA: 1s - loss: 0.0115 - accuracy: 0.9968
1439/1688 [========================>.....] - ETA: 1s - loss: 0.0116 - accuracy: 0.9968
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0115 - accuracy: 0.9968
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0115 - accuracy: 0.9968
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0115 - accuracy: 0.9968
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0115 - accuracy: 0.9968
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0117 - accuracy: 0.9967
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0117 - accuracy: 0.9968
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0117 - accuracy: 0.9968
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0117 - accuracy: 0.9968
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1643/1688 [============================>.] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1655/1688 [============================>.] - ETA: 0s - loss: 0.0115 - accuracy: 0.9968
1667/1688 [============================>.] - ETA: 0s - loss: 0.0116 - accuracy: 0.9968
1679/1688 [============================>.] - ETA: 0s - loss: 0.0115 - accuracy: 0.9968
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0115 - accuracy: 0.9968 - val_loss: 0.0530 - val_accuracy: 0.9875

<keras.callbacks.History object at 0x7f0d825f15b0>

score = model_quantized.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after fine tuning:', score)

Test accuracy after fine tuning: 0.9855999946594238

3. Convert

3.1 Convert to Akida model

When the quantized model produces satisfactory performance, it can be converted to the native Akida format. The convert function returns a model in Akida format ready for inference.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized)
model_akida.summary()

                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          5
______________________________________________

__________________________________________________________________
Layer (type)                        Output shape  Kernel shape

=============== SW/conv2d-dequantizer_2 (Software) ===============

conv2d (InputConv2D)                [13, 13, 32]  (3, 3, 1, 32)
__________________________________________________________________
depthwise_conv2d (DepthwiseConv2D)  [7, 7, 32]    (3, 3, 32, 1)
__________________________________________________________________
conv2d_1 (Conv2D)                   [7, 7, 64]    (1, 1, 32, 64)
__________________________________________________________________
dense (Dense2D)                     [1, 1, 10]    (3136, 10)
__________________________________________________________________
dequantizer_2 (Dequantizer)         [1, 1, 10]    N/A
__________________________________________________________________

3.2. Check performance

accuracy = model_akida.evaluate(x_test, y_test)
print('Test accuracy after conversion:', accuracy)

# For non-regression purposes
assert accuracy > 0.96

Test accuracy after conversion: 0.9865999817848206

3.3 Show predictions for a single image

Display one of the test images, such as the first image in the dataset from above, to visualize the output of the model.

# Test a single example
sample_image = 0
image = x_test[sample_image]
outputs = model_akida.predict(image.reshape(1, 28, 28, 1))
print('Input Label: %i' % y_test[sample_image])

f, axarr = plt.subplots(1, 2)
axarr[0].imshow(x_test[sample_image].reshape((28, 28)), cmap=cm.Greys_r)
axarr[0].set_title('Class %d' % y_test[sample_image])
axarr[1].bar(range(10), outputs.squeeze())
axarr[1].set_xticks(range(10))
plt.show()

print(outputs.squeeze())

Input Label: 7
[ -7.5222554   -5.686741    -3.3973691   -0.09257187 -10.064955
  -5.1028066  -18.955019     7.345481    -3.417158    -0.30249915]

Consider the output from the model above. As is typical in backprop-trained models, the final layer is a Dense layer with one neuron for each of the 10 classes in the dataset. The goal of training is to maximize the response of the neuron corresponding to the label of each training sample while minimizing the responses of the other neurons.

In the bar chart above, you can see the outputs from all 10 neurons. It is easy to see that neuron 7 responds much more strongly than the others. The first sample is indeed a number 7.

4. GXNOR/MNIST

A more robust model called GXNOR/MNIST is provided in the model zoo It is inspired from the GXNOR-Net paper. It comes with its pretrained 2/2/1 helper for which the float training was done for 20 epochs, then the model was then gradually quantized following: 4/4/4 –> 4/4/2 –> 2/2/2 –> 2/2/1 with a 15 epochs QAT step at each quantization stage.