Note
Go to the end to download the full example code.
Global Akida workflow
Using the MNIST dataset, this example shows the definition and training of a keras floating point model, its quantization to 8-bit with the help of calibration, its quantization to 4-bit using QAT and its conversion to Akida. Notice that the performance of the original keras floating point model is maintained throughout the Akida flow. Please refer to the Akida user guide for further information.
Note
Please refer to the TensorFlow tf.keras.models module for model creation/import details and the TensorFlow Guide for TensorFlow usage.
The MNIST example below is light enough so that a GPU is not needed for training.
Global Akida workflow
1. Create and train
1.1. Load and reshape MNIST dataset
import numpy as np
import matplotlib.cm as cm
import matplotlib.pyplot as plt
from keras.datasets import mnist
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Add a channels dimension to the image sets as Akida expects 4-D inputs (corresponding to
# (num_samples, width, height, channels). Note: MNIST is a grayscale dataset and is unusual
# in this respect - most image data already includes a channel dimension, and this step will
# not be necessary.
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
# Display a few images from the test set
f, axarr = plt.subplots(1, 4)
for i in range(0, 4):
axarr[i].imshow(x_test[i].reshape((28, 28)), cmap=cm.Greys_r)
axarr[i].set_title('Class %d' % y_test[i])
plt.show()
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
8192/11490434 [..............................] - ETA: 0s
16384/11490434 [..............................] - ETA: 4:20
24576/11490434 [..............................] - ETA: 3:25
32768/11490434 [..............................] - ETA: 2:59
40960/11490434 [..............................] - ETA: 2:43
49152/11490434 [..............................] - ETA: 3:07
57344/11490434 [..............................] - ETA: 2:52
65536/11490434 [..............................] - ETA: 2:41
73728/11490434 [..............................] - ETA: 2:34
81920/11490434 [..............................] - ETA: 2:45
98304/11490434 [..............................] - ETA: 2:26
114688/11490434 [..............................] - ETA: 2:13
131072/11490434 [..............................] - ETA: 2:03
139264/11490434 [..............................] - ETA: 2:11
155648/11490434 [..............................] - ETA: 2:01
172032/11490434 [..............................] - ETA: 1:53
196608/11490434 [..............................] - ETA: 1:43
221184/11490434 [..............................] - ETA: 1:34
237568/11490434 [..............................] - ETA: 1:34
262144/11490434 [..............................] - ETA: 1:27
286720/11490434 [..............................] - ETA: 1:22
311296/11490434 [..............................] - ETA: 1:18
335872/11490434 [..............................] - ETA: 1:14
368640/11490434 [..............................] - ETA: 1:09
376832/11490434 [..............................] - ETA: 1:10
417792/11490434 [>.............................] - ETA: 1:04
458752/11490434 [>.............................] - ETA: 1:00
491520/11490434 [>.............................] - ETA: 57s
524288/11490434 [>.............................] - ETA: 54s
565248/11490434 [>.............................] - ETA: 51s
606208/11490434 [>.............................] - ETA: 50s
663552/11490434 [>.............................] - ETA: 46s
737280/11490434 [>.............................] - ETA: 42s
802816/11490434 [=>............................] - ETA: 39s
851968/11490434 [=>............................] - ETA: 37s
925696/11490434 [=>............................] - ETA: 34s
991232/11490434 [=>............................] - ETA: 33s
1040384/11490434 [=>............................] - ETA: 32s
1130496/11490434 [=>............................] - ETA: 30s
1245184/11490434 [==>...........................] - ETA: 27s
1359872/11490434 [==>...........................] - ETA: 25s
1458176/11490434 [==>...........................] - ETA: 23s
1589248/11490434 [===>..........................] - ETA: 21s
1638400/11490434 [===>..........................] - ETA: 21s
1720320/11490434 [===>..........................] - ETA: 20s
1884160/11490434 [===>..........................] - ETA: 18s
2031616/11490434 [====>.........................] - ETA: 17s
2211840/11490434 [====>.........................] - ETA: 15s
2408448/11490434 [=====>........................] - ETA: 14s
2605056/11490434 [=====>........................] - ETA: 13s
2752512/11490434 [======>.......................] - ETA: 12s
2916352/11490434 [======>.......................] - ETA: 11s
3211264/11490434 [=======>......................] - ETA: 10s
3440640/11490434 [=======>......................] - ETA: 9s
3670016/11490434 [========>.....................] - ETA: 8s
3997696/11490434 [=========>....................] - ETA: 7s
4358144/11490434 [==========>...................] - ETA: 6s
4669440/11490434 [===========>..................] - ETA: 6s
5144576/11490434 [============>.................] - ETA: 5s
5660672/11490434 [=============>................] - ETA: 4s
6029312/11490434 [==============>...............] - ETA: 4s
6782976/11490434 [================>.............] - ETA: 3s
7536640/11490434 [==================>...........] - ETA: 2s
8159232/11490434 [====================>.........] - ETA: 1s
8847360/11490434 [======================>.......] - ETA: 1s
9273344/11490434 [=======================>......] - ETA: 1s
9912320/11490434 [========================>.....] - ETA: 0s
10551296/11490434 [==========================>...] - ETA: 0s
11157504/11490434 [============================>.] - ETA: 0s
11490434/11490434 [==============================] - 5s 0us/step
1.2. Model definition
Note that at this stage, there is nothing specific to the Akida IP. The model constructed below, as inspired by this example, is a completely standard Keras CNN model.
import keras
model_keras = keras.models.Sequential([
keras.layers.Rescaling(1. / 255, input_shape=(28, 28, 1)),
keras.layers.Conv2D(filters=32, kernel_size=3, strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
# Separable layer
keras.layers.DepthwiseConv2D(kernel_size=3, padding='same', strides=2),
keras.layers.Conv2D(filters=64, kernel_size=1, padding='same'),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
rescaling (Rescaling) (None, 28, 28, 1) 0
conv2d (Conv2D) (None, 13, 13, 32) 320
batch_normalization (Batch (None, 13, 13, 32) 128
Normalization)
re_lu (ReLU) (None, 13, 13, 32) 0
depthwise_conv2d (Depthwis (None, 7, 7, 32) 320
eConv2D)
conv2d_1 (Conv2D) (None, 7, 7, 64) 2112
batch_normalization_1 (Bat (None, 7, 7, 64) 256
chNormalization)
re_lu_1 (ReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34506 (134.79 KB)
Trainable params: 34314 (134.04 KB)
Non-trainable params: 192 (768.00 Byte)
_________________________________________________________________
1.3. Model training
Given the model created above, train the model and check its accuracy. The model should achieve a test accuracy over 98% after 10 epochs.
from keras.optimizers import Adam
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=Adam(learning_rate=1e-3),
metrics=['accuracy'])
_ = model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
Epoch 1/10
1/1688 [..............................] - ETA: 49:23 - loss: 2.4728 - accuracy: 0.0625
23/1688 [..............................] - ETA: 3s - loss: 1.3669 - accuracy: 0.5435
46/1688 [..............................] - ETA: 3s - loss: 0.9708 - accuracy: 0.6855
68/1688 [>.............................] - ETA: 3s - loss: 0.7950 - accuracy: 0.7445
91/1688 [>.............................] - ETA: 3s - loss: 0.6824 - accuracy: 0.7823
114/1688 [=>............................] - ETA: 3s - loss: 0.6177 - accuracy: 0.8032
137/1688 [=>............................] - ETA: 3s - loss: 0.5724 - accuracy: 0.8196
160/1688 [=>............................] - ETA: 3s - loss: 0.5318 - accuracy: 0.8338
183/1688 [==>...........................] - ETA: 3s - loss: 0.5078 - accuracy: 0.8412
207/1688 [==>...........................] - ETA: 3s - loss: 0.4757 - accuracy: 0.8524
230/1688 [===>..........................] - ETA: 3s - loss: 0.4461 - accuracy: 0.8625
254/1688 [===>..........................] - ETA: 3s - loss: 0.4244 - accuracy: 0.8693
276/1688 [===>..........................] - ETA: 3s - loss: 0.4094 - accuracy: 0.8747
299/1688 [====>.........................] - ETA: 3s - loss: 0.3950 - accuracy: 0.8779
322/1688 [====>.........................] - ETA: 3s - loss: 0.3802 - accuracy: 0.8824
345/1688 [=====>........................] - ETA: 3s - loss: 0.3673 - accuracy: 0.8867
369/1688 [=====>........................] - ETA: 2s - loss: 0.3563 - accuracy: 0.8899
392/1688 [=====>........................] - ETA: 2s - loss: 0.3486 - accuracy: 0.8923
415/1688 [======>.......................] - ETA: 2s - loss: 0.3403 - accuracy: 0.8947
438/1688 [======>.......................] - ETA: 2s - loss: 0.3317 - accuracy: 0.8974
461/1688 [=======>......................] - ETA: 2s - loss: 0.3245 - accuracy: 0.8998
484/1688 [=======>......................] - ETA: 2s - loss: 0.3161 - accuracy: 0.9025
507/1688 [========>.....................] - ETA: 2s - loss: 0.3087 - accuracy: 0.9048
530/1688 [========>.....................] - ETA: 2s - loss: 0.2999 - accuracy: 0.9080
553/1688 [========>.....................] - ETA: 2s - loss: 0.2929 - accuracy: 0.9103
576/1688 [=========>....................] - ETA: 2s - loss: 0.2887 - accuracy: 0.9116
599/1688 [=========>....................] - ETA: 2s - loss: 0.2842 - accuracy: 0.9129
622/1688 [==========>...................] - ETA: 2s - loss: 0.2779 - accuracy: 0.9151
645/1688 [==========>...................] - ETA: 2s - loss: 0.2735 - accuracy: 0.9165
668/1688 [==========>...................] - ETA: 2s - loss: 0.2697 - accuracy: 0.9177
691/1688 [===========>..................] - ETA: 2s - loss: 0.2650 - accuracy: 0.9190
714/1688 [===========>..................] - ETA: 2s - loss: 0.2603 - accuracy: 0.9203
738/1688 [============>.................] - ETA: 2s - loss: 0.2557 - accuracy: 0.9220
761/1688 [============>.................] - ETA: 2s - loss: 0.2520 - accuracy: 0.9232
784/1688 [============>.................] - ETA: 2s - loss: 0.2478 - accuracy: 0.9245
807/1688 [=============>................] - ETA: 1s - loss: 0.2435 - accuracy: 0.9257
830/1688 [=============>................] - ETA: 1s - loss: 0.2401 - accuracy: 0.9269
853/1688 [==============>...............] - ETA: 1s - loss: 0.2366 - accuracy: 0.9278
876/1688 [==============>...............] - ETA: 1s - loss: 0.2341 - accuracy: 0.9287
899/1688 [==============>...............] - ETA: 1s - loss: 0.2302 - accuracy: 0.9299
922/1688 [===============>..............] - ETA: 1s - loss: 0.2272 - accuracy: 0.9308
945/1688 [===============>..............] - ETA: 1s - loss: 0.2248 - accuracy: 0.9316
967/1688 [================>.............] - ETA: 1s - loss: 0.2220 - accuracy: 0.9324
990/1688 [================>.............] - ETA: 1s - loss: 0.2187 - accuracy: 0.9334
1013/1688 [=================>............] - ETA: 1s - loss: 0.2164 - accuracy: 0.9340
1036/1688 [=================>............] - ETA: 1s - loss: 0.2134 - accuracy: 0.9348
1059/1688 [=================>............] - ETA: 1s - loss: 0.2115 - accuracy: 0.9356
1082/1688 [==================>...........] - ETA: 1s - loss: 0.2095 - accuracy: 0.9362
1105/1688 [==================>...........] - ETA: 1s - loss: 0.2073 - accuracy: 0.9369
1128/1688 [===================>..........] - ETA: 1s - loss: 0.2062 - accuracy: 0.9371
1150/1688 [===================>..........] - ETA: 1s - loss: 0.2045 - accuracy: 0.9377
1173/1688 [===================>..........] - ETA: 1s - loss: 0.2019 - accuracy: 0.9385
1196/1688 [====================>.........] - ETA: 1s - loss: 0.2003 - accuracy: 0.9390
1219/1688 [====================>.........] - ETA: 1s - loss: 0.1981 - accuracy: 0.9398
1242/1688 [=====================>........] - ETA: 0s - loss: 0.1959 - accuracy: 0.9405
1265/1688 [=====================>........] - ETA: 0s - loss: 0.1938 - accuracy: 0.9413
1288/1688 [=====================>........] - ETA: 0s - loss: 0.1921 - accuracy: 0.9419
1311/1688 [======================>.......] - ETA: 0s - loss: 0.1909 - accuracy: 0.9423
1334/1688 [======================>.......] - ETA: 0s - loss: 0.1896 - accuracy: 0.9427
1357/1688 [=======================>......] - ETA: 0s - loss: 0.1882 - accuracy: 0.9432
1380/1688 [=======================>......] - ETA: 0s - loss: 0.1866 - accuracy: 0.9436
1403/1688 [=======================>......] - ETA: 0s - loss: 0.1852 - accuracy: 0.9440
1426/1688 [========================>.....] - ETA: 0s - loss: 0.1839 - accuracy: 0.9443
1449/1688 [========================>.....] - ETA: 0s - loss: 0.1825 - accuracy: 0.9448
1472/1688 [=========================>....] - ETA: 0s - loss: 0.1817 - accuracy: 0.9451
1493/1688 [=========================>....] - ETA: 0s - loss: 0.1803 - accuracy: 0.9455
1514/1688 [=========================>....] - ETA: 0s - loss: 0.1789 - accuracy: 0.9459
1536/1688 [==========================>...] - ETA: 0s - loss: 0.1776 - accuracy: 0.9462
1557/1688 [==========================>...] - ETA: 0s - loss: 0.1763 - accuracy: 0.9467
1579/1688 [===========================>..] - ETA: 0s - loss: 0.1752 - accuracy: 0.9470
1601/1688 [===========================>..] - ETA: 0s - loss: 0.1736 - accuracy: 0.9475
1623/1688 [===========================>..] - ETA: 0s - loss: 0.1723 - accuracy: 0.9479
1645/1688 [============================>.] - ETA: 0s - loss: 0.1710 - accuracy: 0.9482
1667/1688 [============================>.] - ETA: 0s - loss: 0.1705 - accuracy: 0.9484
1688/1688 [==============================] - ETA: 0s - loss: 0.1693 - accuracy: 0.9487
1688/1688 [==============================] - 6s 3ms/step - loss: 0.1693 - accuracy: 0.9487 - val_loss: 0.0752 - val_accuracy: 0.9777
Epoch 2/10
1/1688 [..............................] - ETA: 3s - loss: 0.0527 - accuracy: 0.9688
24/1688 [..............................] - ETA: 3s - loss: 0.0817 - accuracy: 0.9701
46/1688 [..............................] - ETA: 3s - loss: 0.0801 - accuracy: 0.9742
69/1688 [>.............................] - ETA: 3s - loss: 0.0702 - accuracy: 0.9755
92/1688 [>.............................] - ETA: 3s - loss: 0.0683 - accuracy: 0.9776
115/1688 [=>............................] - ETA: 3s - loss: 0.0656 - accuracy: 0.9783
138/1688 [=>............................] - ETA: 3s - loss: 0.0622 - accuracy: 0.9798
160/1688 [=>............................] - ETA: 3s - loss: 0.0630 - accuracy: 0.9799
183/1688 [==>...........................] - ETA: 3s - loss: 0.0673 - accuracy: 0.9793
206/1688 [==>...........................] - ETA: 3s - loss: 0.0674 - accuracy: 0.9794
229/1688 [===>..........................] - ETA: 3s - loss: 0.0681 - accuracy: 0.9779
253/1688 [===>..........................] - ETA: 3s - loss: 0.0675 - accuracy: 0.9786
275/1688 [===>..........................] - ETA: 3s - loss: 0.0654 - accuracy: 0.9793
298/1688 [====>.........................] - ETA: 3s - loss: 0.0653 - accuracy: 0.9788
321/1688 [====>.........................] - ETA: 3s - loss: 0.0672 - accuracy: 0.9787
344/1688 [=====>........................] - ETA: 3s - loss: 0.0677 - accuracy: 0.9783
367/1688 [=====>........................] - ETA: 2s - loss: 0.0694 - accuracy: 0.9779
390/1688 [=====>........................] - ETA: 2s - loss: 0.0703 - accuracy: 0.9777
413/1688 [======>.......................] - ETA: 2s - loss: 0.0702 - accuracy: 0.9775
436/1688 [======>.......................] - ETA: 2s - loss: 0.0705 - accuracy: 0.9777
459/1688 [=======>......................] - ETA: 2s - loss: 0.0713 - accuracy: 0.9771
482/1688 [=======>......................] - ETA: 2s - loss: 0.0708 - accuracy: 0.9774
505/1688 [=======>......................] - ETA: 2s - loss: 0.0705 - accuracy: 0.9774
528/1688 [========>.....................] - ETA: 2s - loss: 0.0704 - accuracy: 0.9775
552/1688 [========>.....................] - ETA: 2s - loss: 0.0699 - accuracy: 0.9776
575/1688 [=========>....................] - ETA: 2s - loss: 0.0695 - accuracy: 0.9778
598/1688 [=========>....................] - ETA: 2s - loss: 0.0696 - accuracy: 0.9778
621/1688 [==========>...................] - ETA: 2s - loss: 0.0700 - accuracy: 0.9775
644/1688 [==========>...................] - ETA: 2s - loss: 0.0700 - accuracy: 0.9776
666/1688 [==========>...................] - ETA: 2s - loss: 0.0705 - accuracy: 0.9777
689/1688 [===========>..................] - ETA: 2s - loss: 0.0714 - accuracy: 0.9777
712/1688 [===========>..................] - ETA: 2s - loss: 0.0713 - accuracy: 0.9778
734/1688 [============>.................] - ETA: 2s - loss: 0.0713 - accuracy: 0.9779
757/1688 [============>.................] - ETA: 2s - loss: 0.0711 - accuracy: 0.9778
780/1688 [============>.................] - ETA: 2s - loss: 0.0709 - accuracy: 0.9778
802/1688 [=============>................] - ETA: 1s - loss: 0.0718 - accuracy: 0.9776
825/1688 [=============>................] - ETA: 1s - loss: 0.0721 - accuracy: 0.9775
848/1688 [==============>...............] - ETA: 1s - loss: 0.0721 - accuracy: 0.9776
871/1688 [==============>...............] - ETA: 1s - loss: 0.0719 - accuracy: 0.9776
893/1688 [==============>...............] - ETA: 1s - loss: 0.0715 - accuracy: 0.9778
915/1688 [===============>..............] - ETA: 1s - loss: 0.0718 - accuracy: 0.9778
938/1688 [===============>..............] - ETA: 1s - loss: 0.0719 - accuracy: 0.9778
961/1688 [================>.............] - ETA: 1s - loss: 0.0722 - accuracy: 0.9777
984/1688 [================>.............] - ETA: 1s - loss: 0.0713 - accuracy: 0.9780
1007/1688 [================>.............] - ETA: 1s - loss: 0.0709 - accuracy: 0.9782
1030/1688 [=================>............] - ETA: 1s - loss: 0.0703 - accuracy: 0.9783
1052/1688 [=================>............] - ETA: 1s - loss: 0.0704 - accuracy: 0.9783
1075/1688 [==================>...........] - ETA: 1s - loss: 0.0713 - accuracy: 0.9781
1098/1688 [==================>...........] - ETA: 1s - loss: 0.0713 - accuracy: 0.9780
1121/1688 [==================>...........] - ETA: 1s - loss: 0.0713 - accuracy: 0.9781
1144/1688 [===================>..........] - ETA: 1s - loss: 0.0715 - accuracy: 0.9780
1167/1688 [===================>..........] - ETA: 1s - loss: 0.0716 - accuracy: 0.9780
1190/1688 [====================>.........] - ETA: 1s - loss: 0.0714 - accuracy: 0.9780
1213/1688 [====================>.........] - ETA: 1s - loss: 0.0710 - accuracy: 0.9781
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0714 - accuracy: 0.9781
1260/1688 [=====================>........] - ETA: 0s - loss: 0.0708 - accuracy: 0.9782
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0709 - accuracy: 0.9782
1305/1688 [======================>.......] - ETA: 0s - loss: 0.0710 - accuracy: 0.9781
1328/1688 [======================>.......] - ETA: 0s - loss: 0.0711 - accuracy: 0.9782
1350/1688 [======================>.......] - ETA: 0s - loss: 0.0714 - accuracy: 0.9781
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0713 - accuracy: 0.9781
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0710 - accuracy: 0.9782
1418/1688 [========================>.....] - ETA: 0s - loss: 0.0708 - accuracy: 0.9783
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0707 - accuracy: 0.9783
1462/1688 [========================>.....] - ETA: 0s - loss: 0.0707 - accuracy: 0.9784
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0707 - accuracy: 0.9784
1508/1688 [=========================>....] - ETA: 0s - loss: 0.0704 - accuracy: 0.9784
1531/1688 [==========================>...] - ETA: 0s - loss: 0.0708 - accuracy: 0.9783
1553/1688 [==========================>...] - ETA: 0s - loss: 0.0706 - accuracy: 0.9783
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0706 - accuracy: 0.9782
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0702 - accuracy: 0.9783
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0703 - accuracy: 0.9784
1646/1688 [============================>.] - ETA: 0s - loss: 0.0702 - accuracy: 0.9784
1669/1688 [============================>.] - ETA: 0s - loss: 0.0699 - accuracy: 0.9785
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0699 - accuracy: 0.9786 - val_loss: 0.0596 - val_accuracy: 0.9823
Epoch 3/10
1/1688 [..............................] - ETA: 3s - loss: 0.0074 - accuracy: 1.0000
24/1688 [..............................] - ETA: 3s - loss: 0.0410 - accuracy: 0.9870
47/1688 [..............................] - ETA: 3s - loss: 0.0451 - accuracy: 0.9880
69/1688 [>.............................] - ETA: 3s - loss: 0.0419 - accuracy: 0.9887
92/1688 [>.............................] - ETA: 3s - loss: 0.0402 - accuracy: 0.9885
115/1688 [=>............................] - ETA: 3s - loss: 0.0461 - accuracy: 0.9856
138/1688 [=>............................] - ETA: 3s - loss: 0.0436 - accuracy: 0.9866
161/1688 [=>............................] - ETA: 3s - loss: 0.0463 - accuracy: 0.9858
184/1688 [==>...........................] - ETA: 3s - loss: 0.0448 - accuracy: 0.9861
207/1688 [==>...........................] - ETA: 3s - loss: 0.0475 - accuracy: 0.9845
229/1688 [===>..........................] - ETA: 3s - loss: 0.0476 - accuracy: 0.9844
252/1688 [===>..........................] - ETA: 3s - loss: 0.0475 - accuracy: 0.9844
275/1688 [===>..........................] - ETA: 3s - loss: 0.0484 - accuracy: 0.9842
298/1688 [====>.........................] - ETA: 3s - loss: 0.0478 - accuracy: 0.9842
320/1688 [====>.........................] - ETA: 3s - loss: 0.0463 - accuracy: 0.9849
343/1688 [=====>........................] - ETA: 3s - loss: 0.0461 - accuracy: 0.9849
366/1688 [=====>........................] - ETA: 2s - loss: 0.0474 - accuracy: 0.9844
387/1688 [=====>........................] - ETA: 2s - loss: 0.0496 - accuracy: 0.9837
408/1688 [======>.......................] - ETA: 2s - loss: 0.0501 - accuracy: 0.9834
429/1688 [======>.......................] - ETA: 2s - loss: 0.0503 - accuracy: 0.9834
450/1688 [======>.......................] - ETA: 2s - loss: 0.0500 - accuracy: 0.9837
471/1688 [=======>......................] - ETA: 2s - loss: 0.0499 - accuracy: 0.9839
492/1688 [=======>......................] - ETA: 2s - loss: 0.0495 - accuracy: 0.9840
513/1688 [========>.....................] - ETA: 2s - loss: 0.0492 - accuracy: 0.9842
534/1688 [========>.....................] - ETA: 2s - loss: 0.0504 - accuracy: 0.9840
555/1688 [========>.....................] - ETA: 2s - loss: 0.0514 - accuracy: 0.9837
576/1688 [=========>....................] - ETA: 2s - loss: 0.0516 - accuracy: 0.9837
597/1688 [=========>....................] - ETA: 2s - loss: 0.0512 - accuracy: 0.9837
618/1688 [=========>....................] - ETA: 2s - loss: 0.0512 - accuracy: 0.9837
639/1688 [==========>...................] - ETA: 2s - loss: 0.0510 - accuracy: 0.9836
660/1688 [==========>...................] - ETA: 2s - loss: 0.0505 - accuracy: 0.9838
681/1688 [===========>..................] - ETA: 2s - loss: 0.0509 - accuracy: 0.9837
702/1688 [===========>..................] - ETA: 2s - loss: 0.0516 - accuracy: 0.9834
724/1688 [===========>..................] - ETA: 2s - loss: 0.0510 - accuracy: 0.9835
745/1688 [============>.................] - ETA: 2s - loss: 0.0506 - accuracy: 0.9836
767/1688 [============>.................] - ETA: 2s - loss: 0.0503 - accuracy: 0.9837
789/1688 [=============>................] - ETA: 2s - loss: 0.0500 - accuracy: 0.9837
811/1688 [=============>................] - ETA: 2s - loss: 0.0506 - accuracy: 0.9835
833/1688 [=============>................] - ETA: 1s - loss: 0.0504 - accuracy: 0.9836
854/1688 [==============>...............] - ETA: 1s - loss: 0.0504 - accuracy: 0.9837
876/1688 [==============>...............] - ETA: 1s - loss: 0.0504 - accuracy: 0.9838
897/1688 [==============>...............] - ETA: 1s - loss: 0.0502 - accuracy: 0.9839
918/1688 [===============>..............] - ETA: 1s - loss: 0.0507 - accuracy: 0.9837
939/1688 [===============>..............] - ETA: 1s - loss: 0.0508 - accuracy: 0.9838
960/1688 [================>.............] - ETA: 1s - loss: 0.0508 - accuracy: 0.9839
982/1688 [================>.............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9841
1004/1688 [================>.............] - ETA: 1s - loss: 0.0508 - accuracy: 0.9839
1025/1688 [=================>............] - ETA: 1s - loss: 0.0508 - accuracy: 0.9841
1046/1688 [=================>............] - ETA: 1s - loss: 0.0510 - accuracy: 0.9840
1067/1688 [=================>............] - ETA: 1s - loss: 0.0509 - accuracy: 0.9841
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0513 - accuracy: 0.9840
1110/1688 [==================>...........] - ETA: 1s - loss: 0.0514 - accuracy: 0.9838
1131/1688 [===================>..........] - ETA: 1s - loss: 0.0515 - accuracy: 0.9839
1153/1688 [===================>..........] - ETA: 1s - loss: 0.0516 - accuracy: 0.9838
1174/1688 [===================>..........] - ETA: 1s - loss: 0.0520 - accuracy: 0.9836
1195/1688 [====================>.........] - ETA: 1s - loss: 0.0520 - accuracy: 0.9836
1216/1688 [====================>.........] - ETA: 1s - loss: 0.0518 - accuracy: 0.9837
1238/1688 [=====================>........] - ETA: 1s - loss: 0.0515 - accuracy: 0.9837
1259/1688 [=====================>........] - ETA: 1s - loss: 0.0514 - accuracy: 0.9837
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0513 - accuracy: 0.9837
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0511 - accuracy: 0.9837
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0510 - accuracy: 0.9837
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0508 - accuracy: 0.9837
1366/1688 [=======================>......] - ETA: 0s - loss: 0.0509 - accuracy: 0.9838
1387/1688 [=======================>......] - ETA: 0s - loss: 0.0511 - accuracy: 0.9837
1408/1688 [========================>.....] - ETA: 0s - loss: 0.0509 - accuracy: 0.9837
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0508 - accuracy: 0.9838
1450/1688 [========================>.....] - ETA: 0s - loss: 0.0504 - accuracy: 0.9839
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0510 - accuracy: 0.9838
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0512 - accuracy: 0.9837
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0508 - accuracy: 0.9839
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0506 - accuracy: 0.9839
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0504 - accuracy: 0.9840
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0501 - accuracy: 0.9841
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0500 - accuracy: 0.9841
1616/1688 [===========================>..] - ETA: 0s - loss: 0.0499 - accuracy: 0.9841
1636/1688 [============================>.] - ETA: 0s - loss: 0.0498 - accuracy: 0.9842
1656/1688 [============================>.] - ETA: 0s - loss: 0.0499 - accuracy: 0.9841
1676/1688 [============================>.] - ETA: 0s - loss: 0.0500 - accuracy: 0.9841
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0499 - accuracy: 0.9841 - val_loss: 0.0634 - val_accuracy: 0.9830
Epoch 4/10
1/1688 [..............................] - ETA: 4s - loss: 0.0418 - accuracy: 0.9688
21/1688 [..............................] - ETA: 4s - loss: 0.0421 - accuracy: 0.9866
41/1688 [..............................] - ETA: 4s - loss: 0.0292 - accuracy: 0.9909
61/1688 [>.............................] - ETA: 4s - loss: 0.0285 - accuracy: 0.9913
81/1688 [>.............................] - ETA: 4s - loss: 0.0292 - accuracy: 0.9900
101/1688 [>.............................] - ETA: 4s - loss: 0.0320 - accuracy: 0.9901
121/1688 [=>............................] - ETA: 4s - loss: 0.0324 - accuracy: 0.9899
141/1688 [=>............................] - ETA: 3s - loss: 0.0341 - accuracy: 0.9902
161/1688 [=>............................] - ETA: 3s - loss: 0.0339 - accuracy: 0.9901
181/1688 [==>...........................] - ETA: 3s - loss: 0.0356 - accuracy: 0.9891
201/1688 [==>...........................] - ETA: 3s - loss: 0.0370 - accuracy: 0.9885
221/1688 [==>...........................] - ETA: 3s - loss: 0.0360 - accuracy: 0.9888
241/1688 [===>..........................] - ETA: 3s - loss: 0.0367 - accuracy: 0.9886
261/1688 [===>..........................] - ETA: 3s - loss: 0.0363 - accuracy: 0.9889
280/1688 [===>..........................] - ETA: 3s - loss: 0.0359 - accuracy: 0.9888
300/1688 [====>.........................] - ETA: 3s - loss: 0.0366 - accuracy: 0.9886
320/1688 [====>.........................] - ETA: 3s - loss: 0.0376 - accuracy: 0.9885
340/1688 [=====>........................] - ETA: 3s - loss: 0.0369 - accuracy: 0.9889
360/1688 [=====>........................] - ETA: 3s - loss: 0.0364 - accuracy: 0.9890
380/1688 [=====>........................] - ETA: 3s - loss: 0.0361 - accuracy: 0.9888
400/1688 [======>.......................] - ETA: 3s - loss: 0.0362 - accuracy: 0.9889
420/1688 [======>.......................] - ETA: 3s - loss: 0.0353 - accuracy: 0.9891
440/1688 [======>.......................] - ETA: 3s - loss: 0.0363 - accuracy: 0.9888
461/1688 [=======>......................] - ETA: 3s - loss: 0.0354 - accuracy: 0.9891
481/1688 [=======>......................] - ETA: 3s - loss: 0.0356 - accuracy: 0.9889
501/1688 [=======>......................] - ETA: 3s - loss: 0.0363 - accuracy: 0.9889
521/1688 [========>.....................] - ETA: 2s - loss: 0.0357 - accuracy: 0.9890
541/1688 [========>.....................] - ETA: 2s - loss: 0.0359 - accuracy: 0.9889
561/1688 [========>.....................] - ETA: 2s - loss: 0.0359 - accuracy: 0.9889
581/1688 [=========>....................] - ETA: 2s - loss: 0.0362 - accuracy: 0.9889
601/1688 [=========>....................] - ETA: 2s - loss: 0.0365 - accuracy: 0.9888
621/1688 [==========>...................] - ETA: 2s - loss: 0.0362 - accuracy: 0.9889
641/1688 [==========>...................] - ETA: 2s - loss: 0.0364 - accuracy: 0.9889
661/1688 [==========>...................] - ETA: 2s - loss: 0.0364 - accuracy: 0.9889
680/1688 [===========>..................] - ETA: 2s - loss: 0.0365 - accuracy: 0.9889
700/1688 [===========>..................] - ETA: 2s - loss: 0.0361 - accuracy: 0.9890
720/1688 [===========>..................] - ETA: 2s - loss: 0.0356 - accuracy: 0.9892
740/1688 [============>.................] - ETA: 2s - loss: 0.0355 - accuracy: 0.9893
760/1688 [============>.................] - ETA: 2s - loss: 0.0366 - accuracy: 0.9888
780/1688 [============>.................] - ETA: 2s - loss: 0.0363 - accuracy: 0.9890
800/1688 [=============>................] - ETA: 2s - loss: 0.0363 - accuracy: 0.9891
820/1688 [=============>................] - ETA: 2s - loss: 0.0367 - accuracy: 0.9889
840/1688 [=============>................] - ETA: 2s - loss: 0.0366 - accuracy: 0.9889
860/1688 [==============>...............] - ETA: 2s - loss: 0.0368 - accuracy: 0.9888
880/1688 [==============>...............] - ETA: 2s - loss: 0.0373 - accuracy: 0.9885
901/1688 [===============>..............] - ETA: 2s - loss: 0.0370 - accuracy: 0.9886
921/1688 [===============>..............] - ETA: 1s - loss: 0.0372 - accuracy: 0.9885
941/1688 [===============>..............] - ETA: 1s - loss: 0.0378 - accuracy: 0.9884
962/1688 [================>.............] - ETA: 1s - loss: 0.0378 - accuracy: 0.9885
982/1688 [================>.............] - ETA: 1s - loss: 0.0383 - accuracy: 0.9884
1002/1688 [================>.............] - ETA: 1s - loss: 0.0385 - accuracy: 0.9883
1022/1688 [=================>............] - ETA: 1s - loss: 0.0387 - accuracy: 0.9882
1042/1688 [=================>............] - ETA: 1s - loss: 0.0386 - accuracy: 0.9882
1062/1688 [=================>............] - ETA: 1s - loss: 0.0384 - accuracy: 0.9882
1082/1688 [==================>...........] - ETA: 1s - loss: 0.0386 - accuracy: 0.9882
1102/1688 [==================>...........] - ETA: 1s - loss: 0.0390 - accuracy: 0.9881
1122/1688 [==================>...........] - ETA: 1s - loss: 0.0390 - accuracy: 0.9880
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0389 - accuracy: 0.9880
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0388 - accuracy: 0.9880
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0386 - accuracy: 0.9881
1201/1688 [====================>.........] - ETA: 1s - loss: 0.0386 - accuracy: 0.9882
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0388 - accuracy: 0.9881
1241/1688 [=====================>........] - ETA: 1s - loss: 0.0390 - accuracy: 0.9881
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0389 - accuracy: 0.9882
1281/1688 [=====================>........] - ETA: 1s - loss: 0.0389 - accuracy: 0.9880
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0388 - accuracy: 0.9880
1321/1688 [======================>.......] - ETA: 0s - loss: 0.0388 - accuracy: 0.9880
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0389 - accuracy: 0.9880
1361/1688 [=======================>......] - ETA: 0s - loss: 0.0390 - accuracy: 0.9880
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0389 - accuracy: 0.9880
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0390 - accuracy: 0.9880
1421/1688 [========================>.....] - ETA: 0s - loss: 0.0388 - accuracy: 0.9881
1441/1688 [========================>.....] - ETA: 0s - loss: 0.0388 - accuracy: 0.9881
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0391 - accuracy: 0.9880
1481/1688 [=========================>....] - ETA: 0s - loss: 0.0390 - accuracy: 0.9880
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0391 - accuracy: 0.9879
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0392 - accuracy: 0.9879
1541/1688 [==========================>...] - ETA: 0s - loss: 0.0391 - accuracy: 0.9880
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0393 - accuracy: 0.9880
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0393 - accuracy: 0.9880
1601/1688 [===========================>..] - ETA: 0s - loss: 0.0395 - accuracy: 0.9879
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0395 - accuracy: 0.9879
1640/1688 [============================>.] - ETA: 0s - loss: 0.0396 - accuracy: 0.9878
1660/1688 [============================>.] - ETA: 0s - loss: 0.0397 - accuracy: 0.9878
1680/1688 [============================>.] - ETA: 0s - loss: 0.0397 - accuracy: 0.9877
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0399 - accuracy: 0.9877 - val_loss: 0.0662 - val_accuracy: 0.9823
Epoch 5/10
1/1688 [..............................] - ETA: 4s - loss: 0.0247 - accuracy: 1.0000
21/1688 [..............................] - ETA: 4s - loss: 0.0445 - accuracy: 0.9881
41/1688 [..............................] - ETA: 4s - loss: 0.0401 - accuracy: 0.9863
61/1688 [>.............................] - ETA: 4s - loss: 0.0399 - accuracy: 0.9872
81/1688 [>.............................] - ETA: 4s - loss: 0.0376 - accuracy: 0.9865
101/1688 [>.............................] - ETA: 4s - loss: 0.0384 - accuracy: 0.9861
121/1688 [=>............................] - ETA: 4s - loss: 0.0363 - accuracy: 0.9871
141/1688 [=>............................] - ETA: 4s - loss: 0.0363 - accuracy: 0.9871
161/1688 [=>............................] - ETA: 3s - loss: 0.0364 - accuracy: 0.9876
181/1688 [==>...........................] - ETA: 3s - loss: 0.0347 - accuracy: 0.9883
201/1688 [==>...........................] - ETA: 3s - loss: 0.0333 - accuracy: 0.9887
221/1688 [==>...........................] - ETA: 3s - loss: 0.0328 - accuracy: 0.9888
241/1688 [===>..........................] - ETA: 3s - loss: 0.0316 - accuracy: 0.9895
261/1688 [===>..........................] - ETA: 3s - loss: 0.0304 - accuracy: 0.9901
281/1688 [===>..........................] - ETA: 3s - loss: 0.0299 - accuracy: 0.9900
301/1688 [====>.........................] - ETA: 3s - loss: 0.0298 - accuracy: 0.9901
321/1688 [====>.........................] - ETA: 3s - loss: 0.0307 - accuracy: 0.9898
341/1688 [=====>........................] - ETA: 3s - loss: 0.0311 - accuracy: 0.9896
361/1688 [=====>........................] - ETA: 3s - loss: 0.0305 - accuracy: 0.9898
381/1688 [=====>........................] - ETA: 3s - loss: 0.0305 - accuracy: 0.9895
401/1688 [======>.......................] - ETA: 3s - loss: 0.0304 - accuracy: 0.9896
421/1688 [======>.......................] - ETA: 3s - loss: 0.0299 - accuracy: 0.9898
441/1688 [======>.......................] - ETA: 3s - loss: 0.0299 - accuracy: 0.9900
461/1688 [=======>......................] - ETA: 3s - loss: 0.0317 - accuracy: 0.9899
480/1688 [=======>......................] - ETA: 3s - loss: 0.0313 - accuracy: 0.9900
500/1688 [=======>......................] - ETA: 3s - loss: 0.0311 - accuracy: 0.9899
520/1688 [========>.....................] - ETA: 3s - loss: 0.0311 - accuracy: 0.9900
540/1688 [========>.....................] - ETA: 2s - loss: 0.0313 - accuracy: 0.9898
560/1688 [========>.....................] - ETA: 2s - loss: 0.0314 - accuracy: 0.9896
579/1688 [=========>....................] - ETA: 2s - loss: 0.0316 - accuracy: 0.9893
599/1688 [=========>....................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9893
619/1688 [==========>...................] - ETA: 2s - loss: 0.0330 - accuracy: 0.9888
638/1688 [==========>...................] - ETA: 2s - loss: 0.0338 - accuracy: 0.9884
658/1688 [==========>...................] - ETA: 2s - loss: 0.0334 - accuracy: 0.9886
678/1688 [===========>..................] - ETA: 2s - loss: 0.0338 - accuracy: 0.9885
698/1688 [===========>..................] - ETA: 2s - loss: 0.0339 - accuracy: 0.9885
718/1688 [===========>..................] - ETA: 2s - loss: 0.0341 - accuracy: 0.9885
738/1688 [============>.................] - ETA: 2s - loss: 0.0340 - accuracy: 0.9886
758/1688 [============>.................] - ETA: 2s - loss: 0.0338 - accuracy: 0.9886
778/1688 [============>.................] - ETA: 2s - loss: 0.0334 - accuracy: 0.9888
798/1688 [=============>................] - ETA: 2s - loss: 0.0335 - accuracy: 0.9888
818/1688 [=============>................] - ETA: 2s - loss: 0.0334 - accuracy: 0.9890
837/1688 [=============>................] - ETA: 2s - loss: 0.0335 - accuracy: 0.9889
857/1688 [==============>...............] - ETA: 2s - loss: 0.0337 - accuracy: 0.9890
877/1688 [==============>...............] - ETA: 2s - loss: 0.0339 - accuracy: 0.9889
897/1688 [==============>...............] - ETA: 2s - loss: 0.0340 - accuracy: 0.9887
917/1688 [===============>..............] - ETA: 1s - loss: 0.0340 - accuracy: 0.9888
937/1688 [===============>..............] - ETA: 1s - loss: 0.0337 - accuracy: 0.9888
957/1688 [================>.............] - ETA: 1s - loss: 0.0340 - accuracy: 0.9886
977/1688 [================>.............] - ETA: 1s - loss: 0.0342 - accuracy: 0.9886
997/1688 [================>.............] - ETA: 1s - loss: 0.0341 - accuracy: 0.9886
1017/1688 [=================>............] - ETA: 1s - loss: 0.0338 - accuracy: 0.9887
1037/1688 [=================>............] - ETA: 1s - loss: 0.0339 - accuracy: 0.9887
1057/1688 [=================>............] - ETA: 1s - loss: 0.0338 - accuracy: 0.9888
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0335 - accuracy: 0.9889
1097/1688 [==================>...........] - ETA: 1s - loss: 0.0338 - accuracy: 0.9887
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0337 - accuracy: 0.9888
1137/1688 [===================>..........] - ETA: 1s - loss: 0.0337 - accuracy: 0.9888
1156/1688 [===================>..........] - ETA: 1s - loss: 0.0338 - accuracy: 0.9888
1176/1688 [===================>..........] - ETA: 1s - loss: 0.0339 - accuracy: 0.9888
1196/1688 [====================>.........] - ETA: 1s - loss: 0.0338 - accuracy: 0.9888
1216/1688 [====================>.........] - ETA: 1s - loss: 0.0340 - accuracy: 0.9887
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0341 - accuracy: 0.9887
1256/1688 [=====================>........] - ETA: 1s - loss: 0.0339 - accuracy: 0.9888
1276/1688 [=====================>........] - ETA: 1s - loss: 0.0342 - accuracy: 0.9887
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0344 - accuracy: 0.9886
1316/1688 [======================>.......] - ETA: 0s - loss: 0.0346 - accuracy: 0.9886
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0344 - accuracy: 0.9886
1356/1688 [=======================>......] - ETA: 0s - loss: 0.0341 - accuracy: 0.9887
1376/1688 [=======================>......] - ETA: 0s - loss: 0.0341 - accuracy: 0.9888
1396/1688 [=======================>......] - ETA: 0s - loss: 0.0339 - accuracy: 0.9888
1416/1688 [========================>.....] - ETA: 0s - loss: 0.0340 - accuracy: 0.9888
1436/1688 [========================>.....] - ETA: 0s - loss: 0.0338 - accuracy: 0.9888
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0339 - accuracy: 0.9889
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0341 - accuracy: 0.9887
1496/1688 [=========================>....] - ETA: 0s - loss: 0.0340 - accuracy: 0.9888
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0337 - accuracy: 0.9889
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0336 - accuracy: 0.9889
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0336 - accuracy: 0.9889
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0337 - accuracy: 0.9889
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0338 - accuracy: 0.9889
1616/1688 [===========================>..] - ETA: 0s - loss: 0.0339 - accuracy: 0.9889
1636/1688 [============================>.] - ETA: 0s - loss: 0.0339 - accuracy: 0.9889
1656/1688 [============================>.] - ETA: 0s - loss: 0.0340 - accuracy: 0.9889
1676/1688 [============================>.] - ETA: 0s - loss: 0.0341 - accuracy: 0.9888
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0342 - accuracy: 0.9888 - val_loss: 0.0541 - val_accuracy: 0.9857
Epoch 6/10
1/1688 [..............................] - ETA: 3s - loss: 0.0599 - accuracy: 0.9688
24/1688 [..............................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9909
47/1688 [..............................] - ETA: 3s - loss: 0.0175 - accuracy: 0.9920
70/1688 [>.............................] - ETA: 3s - loss: 0.0191 - accuracy: 0.9924
93/1688 [>.............................] - ETA: 3s - loss: 0.0192 - accuracy: 0.9926
116/1688 [=>............................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9925
139/1688 [=>............................] - ETA: 3s - loss: 0.0216 - accuracy: 0.9919
162/1688 [=>............................] - ETA: 3s - loss: 0.0229 - accuracy: 0.9919
185/1688 [==>...........................] - ETA: 3s - loss: 0.0234 - accuracy: 0.9910
207/1688 [==>...........................] - ETA: 3s - loss: 0.0236 - accuracy: 0.9911
231/1688 [===>..........................] - ETA: 3s - loss: 0.0230 - accuracy: 0.9915
254/1688 [===>..........................] - ETA: 3s - loss: 0.0230 - accuracy: 0.9916
276/1688 [===>..........................] - ETA: 3s - loss: 0.0224 - accuracy: 0.9920
299/1688 [====>.........................] - ETA: 3s - loss: 0.0224 - accuracy: 0.9917
322/1688 [====>.........................] - ETA: 3s - loss: 0.0227 - accuracy: 0.9918
345/1688 [=====>........................] - ETA: 3s - loss: 0.0221 - accuracy: 0.9922
368/1688 [=====>........................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9922
390/1688 [=====>........................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9922
413/1688 [======>.......................] - ETA: 2s - loss: 0.0229 - accuracy: 0.9923
436/1688 [======>.......................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9920
459/1688 [=======>......................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9920
482/1688 [=======>......................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9922
505/1688 [=======>......................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9921
528/1688 [========>.....................] - ETA: 2s - loss: 0.0246 - accuracy: 0.9920
551/1688 [========>.....................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9918
574/1688 [=========>....................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9919
596/1688 [=========>....................] - ETA: 2s - loss: 0.0241 - accuracy: 0.9920
619/1688 [==========>...................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9920
641/1688 [==========>...................] - ETA: 2s - loss: 0.0246 - accuracy: 0.9919
664/1688 [==========>...................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9919
687/1688 [===========>..................] - ETA: 2s - loss: 0.0248 - accuracy: 0.9918
710/1688 [===========>..................] - ETA: 2s - loss: 0.0248 - accuracy: 0.9918
733/1688 [============>.................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9918
756/1688 [============>.................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9916
779/1688 [============>.................] - ETA: 2s - loss: 0.0246 - accuracy: 0.9917
802/1688 [=============>................] - ETA: 1s - loss: 0.0254 - accuracy: 0.9915
825/1688 [=============>................] - ETA: 1s - loss: 0.0259 - accuracy: 0.9913
848/1688 [==============>...............] - ETA: 1s - loss: 0.0259 - accuracy: 0.9913
871/1688 [==============>...............] - ETA: 1s - loss: 0.0267 - accuracy: 0.9912
893/1688 [==============>...............] - ETA: 1s - loss: 0.0265 - accuracy: 0.9913
914/1688 [===============>..............] - ETA: 1s - loss: 0.0267 - accuracy: 0.9913
935/1688 [===============>..............] - ETA: 1s - loss: 0.0267 - accuracy: 0.9912
956/1688 [===============>..............] - ETA: 1s - loss: 0.0266 - accuracy: 0.9912
977/1688 [================>.............] - ETA: 1s - loss: 0.0271 - accuracy: 0.9911
998/1688 [================>.............] - ETA: 1s - loss: 0.0270 - accuracy: 0.9911
1019/1688 [=================>............] - ETA: 1s - loss: 0.0266 - accuracy: 0.9912
1040/1688 [=================>............] - ETA: 1s - loss: 0.0268 - accuracy: 0.9911
1062/1688 [=================>............] - ETA: 1s - loss: 0.0269 - accuracy: 0.9911
1083/1688 [==================>...........] - ETA: 1s - loss: 0.0268 - accuracy: 0.9912
1104/1688 [==================>...........] - ETA: 1s - loss: 0.0269 - accuracy: 0.9911
1125/1688 [==================>...........] - ETA: 1s - loss: 0.0272 - accuracy: 0.9911
1146/1688 [===================>..........] - ETA: 1s - loss: 0.0271 - accuracy: 0.9911
1166/1688 [===================>..........] - ETA: 1s - loss: 0.0274 - accuracy: 0.9910
1187/1688 [====================>.........] - ETA: 1s - loss: 0.0274 - accuracy: 0.9910
1209/1688 [====================>.........] - ETA: 1s - loss: 0.0273 - accuracy: 0.9911
1230/1688 [====================>.........] - ETA: 1s - loss: 0.0270 - accuracy: 0.9912
1251/1688 [=====================>........] - ETA: 0s - loss: 0.0268 - accuracy: 0.9912
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0270 - accuracy: 0.9911
1293/1688 [=====================>........] - ETA: 0s - loss: 0.0270 - accuracy: 0.9911
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0270 - accuracy: 0.9911
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0271 - accuracy: 0.9911
1351/1688 [=======================>......] - ETA: 0s - loss: 0.0271 - accuracy: 0.9910
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0273 - accuracy: 0.9909
1388/1688 [=======================>......] - ETA: 0s - loss: 0.0273 - accuracy: 0.9909
1406/1688 [=======================>......] - ETA: 0s - loss: 0.0273 - accuracy: 0.9909
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0276 - accuracy: 0.9908
1442/1688 [========================>.....] - ETA: 0s - loss: 0.0276 - accuracy: 0.9907
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0277 - accuracy: 0.9907
1480/1688 [=========================>....] - ETA: 0s - loss: 0.0278 - accuracy: 0.9908
1498/1688 [=========================>....] - ETA: 0s - loss: 0.0277 - accuracy: 0.9908
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0276 - accuracy: 0.9908
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0277 - accuracy: 0.9908
1554/1688 [==========================>...] - ETA: 0s - loss: 0.0279 - accuracy: 0.9907
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0279 - accuracy: 0.9906
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0280 - accuracy: 0.9906
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0280 - accuracy: 0.9906
1627/1688 [===========================>..] - ETA: 0s - loss: 0.0279 - accuracy: 0.9906
1646/1688 [============================>.] - ETA: 0s - loss: 0.0278 - accuracy: 0.9907
1664/1688 [============================>.] - ETA: 0s - loss: 0.0275 - accuracy: 0.9908
1682/1688 [============================>.] - ETA: 0s - loss: 0.0275 - accuracy: 0.9908
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0276 - accuracy: 0.9908 - val_loss: 0.0689 - val_accuracy: 0.9833
Epoch 7/10
1/1688 [..............................] - ETA: 3s - loss: 0.0019 - accuracy: 1.0000
24/1688 [..............................] - ETA: 3s - loss: 0.0179 - accuracy: 0.9948
47/1688 [..............................] - ETA: 3s - loss: 0.0132 - accuracy: 0.9960
70/1688 [>.............................] - ETA: 3s - loss: 0.0125 - accuracy: 0.9960
93/1688 [>.............................] - ETA: 3s - loss: 0.0140 - accuracy: 0.9960
116/1688 [=>............................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9952
137/1688 [=>............................] - ETA: 3s - loss: 0.0151 - accuracy: 0.9948
158/1688 [=>............................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9949
179/1688 [==>...........................] - ETA: 3s - loss: 0.0152 - accuracy: 0.9948
201/1688 [==>...........................] - ETA: 3s - loss: 0.0153 - accuracy: 0.9947
222/1688 [==>...........................] - ETA: 3s - loss: 0.0168 - accuracy: 0.9945
244/1688 [===>..........................] - ETA: 3s - loss: 0.0163 - accuracy: 0.9947
265/1688 [===>..........................] - ETA: 3s - loss: 0.0172 - accuracy: 0.9946
287/1688 [====>.........................] - ETA: 3s - loss: 0.0179 - accuracy: 0.9946
309/1688 [====>.........................] - ETA: 3s - loss: 0.0180 - accuracy: 0.9944
331/1688 [====>.........................] - ETA: 3s - loss: 0.0183 - accuracy: 0.9943
352/1688 [=====>........................] - ETA: 3s - loss: 0.0181 - accuracy: 0.9943
373/1688 [=====>........................] - ETA: 3s - loss: 0.0180 - accuracy: 0.9943
394/1688 [======>.......................] - ETA: 3s - loss: 0.0177 - accuracy: 0.9944
414/1688 [======>.......................] - ETA: 2s - loss: 0.0177 - accuracy: 0.9944
436/1688 [======>.......................] - ETA: 2s - loss: 0.0175 - accuracy: 0.9944
457/1688 [=======>......................] - ETA: 2s - loss: 0.0175 - accuracy: 0.9943
478/1688 [=======>......................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9943
500/1688 [=======>......................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9940
521/1688 [========>.....................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9939
542/1688 [========>.....................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9937
564/1688 [=========>....................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9935
585/1688 [=========>....................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9933
607/1688 [=========>....................] - ETA: 2s - loss: 0.0197 - accuracy: 0.9931
629/1688 [==========>...................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9932
651/1688 [==========>...................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9932
673/1688 [==========>...................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9930
694/1688 [===========>..................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9930
716/1688 [===========>..................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9931
737/1688 [============>.................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9931
758/1688 [============>.................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9933
779/1688 [============>.................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9932
801/1688 [=============>................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9931
823/1688 [=============>................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9930
845/1688 [==============>...............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9927
867/1688 [==============>...............] - ETA: 1s - loss: 0.0207 - accuracy: 0.9928
889/1688 [==============>...............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9928
910/1688 [===============>..............] - ETA: 1s - loss: 0.0203 - accuracy: 0.9930
930/1688 [===============>..............] - ETA: 1s - loss: 0.0202 - accuracy: 0.9930
950/1688 [===============>..............] - ETA: 1s - loss: 0.0201 - accuracy: 0.9931
970/1688 [================>.............] - ETA: 1s - loss: 0.0202 - accuracy: 0.9931
990/1688 [================>.............] - ETA: 1s - loss: 0.0201 - accuracy: 0.9932
1010/1688 [================>.............] - ETA: 1s - loss: 0.0200 - accuracy: 0.9932
1030/1688 [=================>............] - ETA: 1s - loss: 0.0200 - accuracy: 0.9932
1050/1688 [=================>............] - ETA: 1s - loss: 0.0203 - accuracy: 0.9931
1069/1688 [=================>............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9931
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0207 - accuracy: 0.9931
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0206 - accuracy: 0.9931
1129/1688 [===================>..........] - ETA: 1s - loss: 0.0210 - accuracy: 0.9930
1149/1688 [===================>..........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9929
1169/1688 [===================>..........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9927
1189/1688 [====================>.........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9927
1209/1688 [====================>.........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9926
1229/1688 [====================>.........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9927
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9927
1267/1688 [=====================>........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9927
1287/1688 [=====================>........] - ETA: 0s - loss: 0.0214 - accuracy: 0.9928
1307/1688 [======================>.......] - ETA: 0s - loss: 0.0215 - accuracy: 0.9928
1327/1688 [======================>.......] - ETA: 0s - loss: 0.0215 - accuracy: 0.9929
1347/1688 [======================>.......] - ETA: 0s - loss: 0.0217 - accuracy: 0.9928
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1387/1688 [=======================>......] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1407/1688 [========================>.....] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9928
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0221 - accuracy: 0.9926
1507/1688 [=========================>....] - ETA: 0s - loss: 0.0220 - accuracy: 0.9926
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0222 - accuracy: 0.9926
1546/1688 [==========================>...] - ETA: 0s - loss: 0.0224 - accuracy: 0.9925
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0226 - accuracy: 0.9924
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0230 - accuracy: 0.9923
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1646/1688 [============================>.] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1666/1688 [============================>.] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1687/1688 [============================>.] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0231 - accuracy: 0.9923 - val_loss: 0.0655 - val_accuracy: 0.9843
Epoch 8/10
1/1688 [..............................] - ETA: 4s - loss: 0.0086 - accuracy: 1.0000
23/1688 [..............................] - ETA: 3s - loss: 0.0201 - accuracy: 0.9932
45/1688 [..............................] - ETA: 3s - loss: 0.0178 - accuracy: 0.9944
66/1688 [>.............................] - ETA: 3s - loss: 0.0210 - accuracy: 0.9929
87/1688 [>.............................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9935
107/1688 [>.............................] - ETA: 3s - loss: 0.0176 - accuracy: 0.9939
127/1688 [=>............................] - ETA: 3s - loss: 0.0164 - accuracy: 0.9946
147/1688 [=>............................] - ETA: 3s - loss: 0.0146 - accuracy: 0.9953
167/1688 [=>............................] - ETA: 3s - loss: 0.0153 - accuracy: 0.9951
187/1688 [==>...........................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9955
207/1688 [==>...........................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9956
227/1688 [===>..........................] - ETA: 3s - loss: 0.0139 - accuracy: 0.9959
248/1688 [===>..........................] - ETA: 3s - loss: 0.0134 - accuracy: 0.9960
268/1688 [===>..........................] - ETA: 3s - loss: 0.0132 - accuracy: 0.9962
288/1688 [====>.........................] - ETA: 3s - loss: 0.0139 - accuracy: 0.9960
309/1688 [====>.........................] - ETA: 3s - loss: 0.0146 - accuracy: 0.9957
329/1688 [====>.........................] - ETA: 3s - loss: 0.0141 - accuracy: 0.9956
350/1688 [=====>........................] - ETA: 3s - loss: 0.0137 - accuracy: 0.9959
370/1688 [=====>........................] - ETA: 3s - loss: 0.0133 - accuracy: 0.9961
390/1688 [=====>........................] - ETA: 3s - loss: 0.0133 - accuracy: 0.9961
411/1688 [======>.......................] - ETA: 3s - loss: 0.0131 - accuracy: 0.9960
432/1688 [======>.......................] - ETA: 3s - loss: 0.0134 - accuracy: 0.9959
452/1688 [=======>......................] - ETA: 3s - loss: 0.0132 - accuracy: 0.9960
473/1688 [=======>......................] - ETA: 3s - loss: 0.0138 - accuracy: 0.9958
494/1688 [=======>......................] - ETA: 2s - loss: 0.0137 - accuracy: 0.9958
515/1688 [========>.....................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9959
536/1688 [========>.....................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9959
557/1688 [========>.....................] - ETA: 2s - loss: 0.0137 - accuracy: 0.9959
577/1688 [=========>....................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9958
598/1688 [=========>....................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9958
619/1688 [==========>...................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9957
640/1688 [==========>...................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9954
661/1688 [==========>...................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9955
681/1688 [===========>..................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9953
701/1688 [===========>..................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9954
722/1688 [===========>..................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9953
743/1688 [============>.................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9952
763/1688 [============>.................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9951
783/1688 [============>.................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9951
803/1688 [=============>................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9950
824/1688 [=============>................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9950
844/1688 [==============>...............] - ETA: 2s - loss: 0.0161 - accuracy: 0.9950
864/1688 [==============>...............] - ETA: 2s - loss: 0.0161 - accuracy: 0.9951
884/1688 [==============>...............] - ETA: 2s - loss: 0.0160 - accuracy: 0.9951
904/1688 [===============>..............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9949
924/1688 [===============>..............] - ETA: 1s - loss: 0.0168 - accuracy: 0.9947
945/1688 [===============>..............] - ETA: 1s - loss: 0.0168 - accuracy: 0.9947
965/1688 [================>.............] - ETA: 1s - loss: 0.0169 - accuracy: 0.9946
985/1688 [================>.............] - ETA: 1s - loss: 0.0168 - accuracy: 0.9947
1005/1688 [================>.............] - ETA: 1s - loss: 0.0167 - accuracy: 0.9947
1025/1688 [=================>............] - ETA: 1s - loss: 0.0168 - accuracy: 0.9947
1046/1688 [=================>............] - ETA: 1s - loss: 0.0167 - accuracy: 0.9947
1066/1688 [=================>............] - ETA: 1s - loss: 0.0169 - accuracy: 0.9946
1087/1688 [==================>...........] - ETA: 1s - loss: 0.0172 - accuracy: 0.9946
1107/1688 [==================>...........] - ETA: 1s - loss: 0.0173 - accuracy: 0.9946
1128/1688 [===================>..........] - ETA: 1s - loss: 0.0173 - accuracy: 0.9946
1148/1688 [===================>..........] - ETA: 1s - loss: 0.0175 - accuracy: 0.9945
1169/1688 [===================>..........] - ETA: 1s - loss: 0.0175 - accuracy: 0.9945
1190/1688 [====================>.........] - ETA: 1s - loss: 0.0175 - accuracy: 0.9945
1210/1688 [====================>.........] - ETA: 1s - loss: 0.0176 - accuracy: 0.9945
1230/1688 [====================>.........] - ETA: 1s - loss: 0.0176 - accuracy: 0.9944
1251/1688 [=====================>........] - ETA: 1s - loss: 0.0178 - accuracy: 0.9944
1271/1688 [=====================>........] - ETA: 1s - loss: 0.0178 - accuracy: 0.9943
1291/1688 [=====================>........] - ETA: 0s - loss: 0.0184 - accuracy: 0.9942
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0186 - accuracy: 0.9943
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0186 - accuracy: 0.9943
1352/1688 [=======================>......] - ETA: 0s - loss: 0.0187 - accuracy: 0.9942
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0190 - accuracy: 0.9941
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0188 - accuracy: 0.9941
1412/1688 [========================>.....] - ETA: 0s - loss: 0.0187 - accuracy: 0.9942
1433/1688 [========================>.....] - ETA: 0s - loss: 0.0185 - accuracy: 0.9942
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0184 - accuracy: 0.9942
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0185 - accuracy: 0.9942
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0186 - accuracy: 0.9942
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0185 - accuracy: 0.9942
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0185 - accuracy: 0.9942
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0185 - accuracy: 0.9942
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0186 - accuracy: 0.9942
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0185 - accuracy: 0.9942
1616/1688 [===========================>..] - ETA: 0s - loss: 0.0186 - accuracy: 0.9942
1637/1688 [============================>.] - ETA: 0s - loss: 0.0185 - accuracy: 0.9942
1658/1688 [============================>.] - ETA: 0s - loss: 0.0184 - accuracy: 0.9943
1679/1688 [============================>.] - ETA: 0s - loss: 0.0184 - accuracy: 0.9942
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0184 - accuracy: 0.9943 - val_loss: 0.0693 - val_accuracy: 0.9850
Epoch 9/10
1/1688 [..............................] - ETA: 3s - loss: 0.0121 - accuracy: 1.0000
24/1688 [..............................] - ETA: 3s - loss: 0.0235 - accuracy: 0.9922
47/1688 [..............................] - ETA: 3s - loss: 0.0175 - accuracy: 0.9940
70/1688 [>.............................] - ETA: 3s - loss: 0.0138 - accuracy: 0.9955
93/1688 [>.............................] - ETA: 3s - loss: 0.0178 - accuracy: 0.9940
116/1688 [=>............................] - ETA: 3s - loss: 0.0198 - accuracy: 0.9933
139/1688 [=>............................] - ETA: 3s - loss: 0.0194 - accuracy: 0.9930
162/1688 [=>............................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9932
185/1688 [==>...........................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9936
208/1688 [==>...........................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9932
231/1688 [===>..........................] - ETA: 3s - loss: 0.0201 - accuracy: 0.9934
254/1688 [===>..........................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9936
277/1688 [===>..........................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9939
299/1688 [====>.........................] - ETA: 3s - loss: 0.0186 - accuracy: 0.9940
320/1688 [====>.........................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9940
341/1688 [=====>........................] - ETA: 3s - loss: 0.0176 - accuracy: 0.9944
362/1688 [=====>........................] - ETA: 3s - loss: 0.0171 - accuracy: 0.9945
384/1688 [=====>........................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9945
405/1688 [======>.......................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9946
425/1688 [======>.......................] - ETA: 2s - loss: 0.0167 - accuracy: 0.9947
445/1688 [======>.......................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9947
465/1688 [=======>......................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9949
485/1688 [=======>......................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9951
504/1688 [=======>......................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9952
524/1688 [========>.....................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9952
544/1688 [========>.....................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9952
564/1688 [=========>....................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9950
584/1688 [=========>....................] - ETA: 2s - loss: 0.0158 - accuracy: 0.9951
604/1688 [=========>....................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9951
624/1688 [==========>...................] - ETA: 2s - loss: 0.0156 - accuracy: 0.9951
643/1688 [==========>...................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9952
662/1688 [==========>...................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9951
681/1688 [===========>..................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9950
701/1688 [===========>..................] - ETA: 2s - loss: 0.0156 - accuracy: 0.9951
721/1688 [===========>..................] - ETA: 2s - loss: 0.0158 - accuracy: 0.9949
741/1688 [============>.................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9949
760/1688 [============>.................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9949
780/1688 [============>.................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9948
800/1688 [=============>................] - ETA: 2s - loss: 0.0158 - accuracy: 0.9949
820/1688 [=============>................] - ETA: 2s - loss: 0.0158 - accuracy: 0.9949
840/1688 [=============>................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9950
859/1688 [==============>...............] - ETA: 2s - loss: 0.0157 - accuracy: 0.9951
879/1688 [==============>...............] - ETA: 1s - loss: 0.0157 - accuracy: 0.9950
898/1688 [==============>...............] - ETA: 1s - loss: 0.0156 - accuracy: 0.9951
916/1688 [===============>..............] - ETA: 1s - loss: 0.0155 - accuracy: 0.9951
934/1688 [===============>..............] - ETA: 1s - loss: 0.0155 - accuracy: 0.9951
952/1688 [===============>..............] - ETA: 1s - loss: 0.0154 - accuracy: 0.9952
971/1688 [================>.............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9952
989/1688 [================>.............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9951
1007/1688 [================>.............] - ETA: 1s - loss: 0.0154 - accuracy: 0.9952
1026/1688 [=================>............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9952
1044/1688 [=================>............] - ETA: 1s - loss: 0.0156 - accuracy: 0.9951
1062/1688 [=================>............] - ETA: 1s - loss: 0.0159 - accuracy: 0.9950
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0160 - accuracy: 0.9949
1098/1688 [==================>...........] - ETA: 1s - loss: 0.0162 - accuracy: 0.9949
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0161 - accuracy: 0.9949
1135/1688 [===================>..........] - ETA: 1s - loss: 0.0160 - accuracy: 0.9949
1154/1688 [===================>..........] - ETA: 1s - loss: 0.0161 - accuracy: 0.9949
1172/1688 [===================>..........] - ETA: 1s - loss: 0.0162 - accuracy: 0.9949
1191/1688 [====================>.........] - ETA: 1s - loss: 0.0161 - accuracy: 0.9949
1210/1688 [====================>.........] - ETA: 1s - loss: 0.0162 - accuracy: 0.9949
1228/1688 [====================>.........] - ETA: 1s - loss: 0.0161 - accuracy: 0.9949
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0161 - accuracy: 0.9949
1265/1688 [=====================>........] - ETA: 1s - loss: 0.0161 - accuracy: 0.9950
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0160 - accuracy: 0.9950
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0160 - accuracy: 0.9950
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0158 - accuracy: 0.9950
1338/1688 [======================>.......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9950
1356/1688 [=======================>......] - ETA: 0s - loss: 0.0158 - accuracy: 0.9950
1375/1688 [=======================>......] - ETA: 0s - loss: 0.0162 - accuracy: 0.9949
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0162 - accuracy: 0.9949
1412/1688 [========================>.....] - ETA: 0s - loss: 0.0162 - accuracy: 0.9949
1430/1688 [========================>.....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9947
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0166 - accuracy: 0.9947
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0166 - accuracy: 0.9947
1486/1688 [=========================>....] - ETA: 0s - loss: 0.0165 - accuracy: 0.9947
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0166 - accuracy: 0.9947
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0168 - accuracy: 0.9946
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0171 - accuracy: 0.9945
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0172 - accuracy: 0.9944
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0171 - accuracy: 0.9944
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0170 - accuracy: 0.9945
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0171 - accuracy: 0.9945
1632/1688 [============================>.] - ETA: 0s - loss: 0.0174 - accuracy: 0.9944
1650/1688 [============================>.] - ETA: 0s - loss: 0.0176 - accuracy: 0.9943
1669/1688 [============================>.] - ETA: 0s - loss: 0.0176 - accuracy: 0.9943
1688/1688 [==============================] - ETA: 0s - loss: 0.0177 - accuracy: 0.9943
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0177 - accuracy: 0.9943 - val_loss: 0.0703 - val_accuracy: 0.9843
Epoch 10/10
1/1688 [..............................] - ETA: 4s - loss: 0.0307 - accuracy: 1.0000
19/1688 [..............................] - ETA: 4s - loss: 0.0122 - accuracy: 0.9967
37/1688 [..............................] - ETA: 4s - loss: 0.0122 - accuracy: 0.9958
55/1688 [..............................] - ETA: 4s - loss: 0.0112 - accuracy: 0.9960
74/1688 [>.............................] - ETA: 4s - loss: 0.0144 - accuracy: 0.9945
92/1688 [>.............................] - ETA: 4s - loss: 0.0124 - accuracy: 0.9956
110/1688 [>.............................] - ETA: 4s - loss: 0.0132 - accuracy: 0.9955
128/1688 [=>............................] - ETA: 4s - loss: 0.0122 - accuracy: 0.9958
147/1688 [=>............................] - ETA: 4s - loss: 0.0116 - accuracy: 0.9962
165/1688 [=>............................] - ETA: 4s - loss: 0.0110 - accuracy: 0.9966
183/1688 [==>...........................] - ETA: 4s - loss: 0.0111 - accuracy: 0.9964
201/1688 [==>...........................] - ETA: 4s - loss: 0.0108 - accuracy: 0.9966
219/1688 [==>...........................] - ETA: 4s - loss: 0.0103 - accuracy: 0.9967
237/1688 [===>..........................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9970
256/1688 [===>..........................] - ETA: 3s - loss: 0.0093 - accuracy: 0.9972
274/1688 [===>..........................] - ETA: 3s - loss: 0.0091 - accuracy: 0.9973
292/1688 [====>.........................] - ETA: 3s - loss: 0.0090 - accuracy: 0.9973
310/1688 [====>.........................] - ETA: 3s - loss: 0.0091 - accuracy: 0.9972
329/1688 [====>.........................] - ETA: 3s - loss: 0.0090 - accuracy: 0.9972
347/1688 [=====>........................] - ETA: 3s - loss: 0.0092 - accuracy: 0.9970
366/1688 [=====>........................] - ETA: 3s - loss: 0.0096 - accuracy: 0.9969
385/1688 [=====>........................] - ETA: 3s - loss: 0.0097 - accuracy: 0.9968
403/1688 [======>.......................] - ETA: 3s - loss: 0.0103 - accuracy: 0.9966
421/1688 [======>.......................] - ETA: 3s - loss: 0.0100 - accuracy: 0.9967
440/1688 [======>.......................] - ETA: 3s - loss: 0.0100 - accuracy: 0.9967
458/1688 [=======>......................] - ETA: 3s - loss: 0.0100 - accuracy: 0.9967
477/1688 [=======>......................] - ETA: 3s - loss: 0.0103 - accuracy: 0.9967
496/1688 [=======>......................] - ETA: 3s - loss: 0.0102 - accuracy: 0.9967
514/1688 [========>.....................] - ETA: 3s - loss: 0.0102 - accuracy: 0.9967
532/1688 [========>.....................] - ETA: 3s - loss: 0.0103 - accuracy: 0.9967
551/1688 [========>.....................] - ETA: 3s - loss: 0.0101 - accuracy: 0.9968
570/1688 [=========>....................] - ETA: 3s - loss: 0.0101 - accuracy: 0.9967
589/1688 [=========>....................] - ETA: 3s - loss: 0.0102 - accuracy: 0.9967
607/1688 [=========>....................] - ETA: 3s - loss: 0.0104 - accuracy: 0.9967
626/1688 [==========>...................] - ETA: 2s - loss: 0.0103 - accuracy: 0.9967
644/1688 [==========>...................] - ETA: 2s - loss: 0.0102 - accuracy: 0.9967
663/1688 [==========>...................] - ETA: 2s - loss: 0.0103 - accuracy: 0.9967
681/1688 [===========>..................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9966
699/1688 [===========>..................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9965
718/1688 [===========>..................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9966
737/1688 [============>.................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9965
755/1688 [============>.................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9965
773/1688 [============>.................] - ETA: 2s - loss: 0.0107 - accuracy: 0.9965
792/1688 [=============>................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9964
811/1688 [=============>................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9963
830/1688 [=============>................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9962
848/1688 [==============>...............] - ETA: 2s - loss: 0.0115 - accuracy: 0.9962
866/1688 [==============>...............] - ETA: 2s - loss: 0.0115 - accuracy: 0.9962
884/1688 [==============>...............] - ETA: 2s - loss: 0.0117 - accuracy: 0.9961
903/1688 [===============>..............] - ETA: 2s - loss: 0.0118 - accuracy: 0.9960
921/1688 [===============>..............] - ETA: 2s - loss: 0.0122 - accuracy: 0.9959
940/1688 [===============>..............] - ETA: 2s - loss: 0.0122 - accuracy: 0.9958
959/1688 [================>.............] - ETA: 2s - loss: 0.0123 - accuracy: 0.9958
978/1688 [================>.............] - ETA: 1s - loss: 0.0121 - accuracy: 0.9959
996/1688 [================>.............] - ETA: 1s - loss: 0.0122 - accuracy: 0.9959
1014/1688 [=================>............] - ETA: 1s - loss: 0.0126 - accuracy: 0.9958
1032/1688 [=================>............] - ETA: 1s - loss: 0.0128 - accuracy: 0.9957
1050/1688 [=================>............] - ETA: 1s - loss: 0.0128 - accuracy: 0.9957
1068/1688 [=================>............] - ETA: 1s - loss: 0.0129 - accuracy: 0.9957
1086/1688 [==================>...........] - ETA: 1s - loss: 0.0128 - accuracy: 0.9957
1104/1688 [==================>...........] - ETA: 1s - loss: 0.0129 - accuracy: 0.9957
1122/1688 [==================>...........] - ETA: 1s - loss: 0.0130 - accuracy: 0.9957
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0128 - accuracy: 0.9957
1160/1688 [===================>..........] - ETA: 1s - loss: 0.0127 - accuracy: 0.9958
1179/1688 [===================>..........] - ETA: 1s - loss: 0.0128 - accuracy: 0.9957
1198/1688 [====================>.........] - ETA: 1s - loss: 0.0130 - accuracy: 0.9957
1216/1688 [====================>.........] - ETA: 1s - loss: 0.0132 - accuracy: 0.9957
1234/1688 [====================>.........] - ETA: 1s - loss: 0.0131 - accuracy: 0.9957
1253/1688 [=====================>........] - ETA: 1s - loss: 0.0132 - accuracy: 0.9956
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0132 - accuracy: 0.9956
1291/1688 [=====================>........] - ETA: 1s - loss: 0.0133 - accuracy: 0.9955
1310/1688 [======================>.......] - ETA: 1s - loss: 0.0137 - accuracy: 0.9954
1329/1688 [======================>.......] - ETA: 0s - loss: 0.0137 - accuracy: 0.9954
1348/1688 [======================>.......] - ETA: 0s - loss: 0.0140 - accuracy: 0.9953
1366/1688 [=======================>......] - ETA: 0s - loss: 0.0140 - accuracy: 0.9952
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0141 - accuracy: 0.9952
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0143 - accuracy: 0.9951
1422/1688 [========================>.....] - ETA: 0s - loss: 0.0144 - accuracy: 0.9950
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0144 - accuracy: 0.9951
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1531/1688 [==========================>...] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0147 - accuracy: 0.9950
1642/1688 [============================>.] - ETA: 0s - loss: 0.0147 - accuracy: 0.9950
1661/1688 [============================>.] - ETA: 0s - loss: 0.0147 - accuracy: 0.9951
1680/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9950
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0148 - accuracy: 0.9950 - val_loss: 0.0556 - val_accuracy: 0.9870
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Test accuracy: 0.9846000075340271
2. Quantize
2.1. 8-bit quantization
An Akida accelerator processes 8 or 4-bits integer activations and weights. Therefore, the floating point Keras model must be quantized in preparation to run on an Akida accelerator.
The QuantizeML quantize function can be used to quantize a Keras model for Akida. For this step in this example, an “8/8/8” quantization scheme will be applied to the floating point Keras model to produce 8-bit weights in the first layer, 8-bit weights in all other layers, and 8-bit activations.
The quantization process results in a Keras model with custom QuantizeML quantized layers substituted for the original Keras layers.
All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.
Note
The quantize function applies several transformations to
the original model. For example, it folds the batch normalization layers into the
corresponding neural layers. The new weights are computed according to this folding
operation.
from quantizeml.models import quantize, QuantizationParams
qparams = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)
model_quantized = quantize(model_keras, qparams=qparams)
model_quantized.summary()
/usr/local/lib/python3.11/dist-packages/onnxscript/converter.py:816: FutureWarning: 'onnxscript.values.Op.param_schemas' is deprecated in version 0.1 and will be removed in the future. Please use '.op_signature' instead.
param_schemas = callee.param_schemas()
/usr/local/lib/python3.11/dist-packages/quantizeml/models/quantize.py:479: UserWarning: Quantizing per-axis with random calibration samples is not accurate. Set QuantizationParams.per_tensor_activations=True when calibrating with random samples.
warnings.warn("Quantizing per-axis with random calibration samples is not accurate.\
1/1024 [..............................] - ETA: 3:08
57/1024 [>.............................] - ETA: 0s
114/1024 [==>...........................] - ETA: 0s
172/1024 [====>.........................] - ETA: 0s
230/1024 [=====>........................] - ETA: 0s
288/1024 [=======>......................] - ETA: 0s
345/1024 [=========>....................] - ETA: 0s
402/1024 [==========>...................] - ETA: 0s
459/1024 [============>.................] - ETA: 0s
517/1024 [==============>...............] - ETA: 0s
575/1024 [===============>..............] - ETA: 0s
632/1024 [=================>............] - ETA: 0s
690/1024 [===================>..........] - ETA: 0s
747/1024 [====================>.........] - ETA: 0s
805/1024 [======================>.......] - ETA: 0s
863/1024 [========================>.....] - ETA: 0s
920/1024 [=========================>....] - ETA: 0s
976/1024 [===========================>..] - ETA: 0s
1024/1024 [==============================] - 1s 879us/step
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
rescaling_input (InputLaye [(None, 28, 28, 1)] 0
r)
rescaling (QuantizedRescal (None, 28, 28, 1) 0
ing)
conv2d (QuantizedConv2D) (None, 13, 13, 32) 320
re_lu (QuantizedReLU) (None, 13, 13, 32) 64
depthwise_conv2d (Quantize (None, 7, 7, 32) 384
dDepthwiseConv2D)
conv2d_1 (QuantizedConv2D) (None, 7, 7, 64) 2112
re_lu_1 (QuantizedReLU) (None, 7, 7, 64) 128
flatten (QuantizedFlatten) (None, 3136) 0
dense (QuantizedDense) (None, 10) 31370
dequantizer (Dequantizer) (None, 10) 0
=================================================================
Total params: 34378 (134.29 KB)
Trainable params: 34122 (133.29 KB)
Non-trainable params: 256 (1.00 KB)
_________________________________________________________________
Note
Note that the number of parameters for the floating and quantized models differs, a consequence of the BatchNormalization folding and the additional parameters added for quantization. For further details, please refer to their respective summary.
Check the quantized model accuracy.
def compile_evaluate(model):
""" Compiles and evaluates the model, then return accuracy score. """
model.compile(metrics=['accuracy'])
return model.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after 8-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 8-bit quantization: 0.9801999926567078
2.2. Effect of calibration
The previous call to quantize was made with random samples for calibration
(default parameters). While the observed drop in accuracy is minimal, that is
around 1%, it can be worse on more complex models. Therefore, it is advised to
use a set of real samples from the training set for calibration during a call
to quantize.
Note that this remains a calibration step rather than a training step in that
no output labels are required. Furthermore, any relevant data could be used for
calibration. The recommended settings for calibration that are widely used to
obtain the zoo performance are:
1024 samples
a batch size of 100
2 epochs
model_quantized = quantize(model_keras, qparams=qparams,
samples=x_train, num_samples=1024, batch_size=100, epochs=2)
1/11 [=>............................] - ETA: 3s
11/11 [==============================] - 0s 995us/step
1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 950us/step
Check the accuracy for the quantized and calibrated model.
print('Test accuracy after calibration:', compile_evaluate(model_quantized))
Test accuracy after calibration: 0.984000027179718
Calibrating with real samples on this model recovers the initial float accuracy.
2.3. 4-bit quantization
The accuracy of the 8/8/8 quantized model is equal to that of the Keras floating point model. In some cases, a smaller memory size for the model is required. This can be accomplished through quantization of the model to smaller bitwidths.
The model will now be quantized to 8/4/4, that is 8-bit weights in the first layer with 4-bit weights and activations in all other layers. Such a quantization scheme will usually introduce a performance drop.
qparams = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)
model_quantized = quantize(model_keras, qparams=qparams,
samples=x_train, num_samples=1024, batch_size=100, epochs=2)
1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step
1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 1ms/step
Check the 4-bit quantized accuracy.
print('Test accuracy after 4-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 4-bit quantization: 0.9839000105857849
2.4. Model fine tuning (Quantization Aware Training)
When a model suffers from an accuracy drop after quantization, fine tuning or Quantization Aware Training (QAT) may recover some or all of the original performance.
Note that since this is a fine tuning step, both the number of epochs and learning rate are expected to be lower than during the initial float training.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=Adam(learning_rate=1e-4),
metrics=['accuracy'])
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
Epoch 1/5
1/1688 [..............................] - ETA: 2:05:02 - loss: 0.0032 - accuracy: 1.0000
11/1688 [..............................] - ETA: 8s - loss: 0.0116 - accuracy: 0.9972
22/1688 [..............................] - ETA: 8s - loss: 0.0095 - accuracy: 0.9972
33/1688 [..............................] - ETA: 8s - loss: 0.0087 - accuracy: 0.9981
44/1688 [..............................] - ETA: 8s - loss: 0.0099 - accuracy: 0.9972
55/1688 [..............................] - ETA: 8s - loss: 0.0089 - accuracy: 0.9977
66/1688 [>.............................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9972
77/1688 [>.............................] - ETA: 8s - loss: 0.0106 - accuracy: 0.9968
88/1688 [>.............................] - ETA: 7s - loss: 0.0108 - accuracy: 0.9964
99/1688 [>.............................] - ETA: 7s - loss: 0.0114 - accuracy: 0.9962
109/1688 [>.............................] - ETA: 7s - loss: 0.0107 - accuracy: 0.9966
119/1688 [=>............................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9966
130/1688 [=>............................] - ETA: 7s - loss: 0.0108 - accuracy: 0.9962
141/1688 [=>............................] - ETA: 7s - loss: 0.0113 - accuracy: 0.9962
152/1688 [=>............................] - ETA: 7s - loss: 0.0108 - accuracy: 0.9965
163/1688 [=>............................] - ETA: 7s - loss: 0.0104 - accuracy: 0.9967
173/1688 [==>...........................] - ETA: 7s - loss: 0.0100 - accuracy: 0.9969
184/1688 [==>...........................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9969
195/1688 [==>...........................] - ETA: 7s - loss: 0.0098 - accuracy: 0.9971
206/1688 [==>...........................] - ETA: 7s - loss: 0.0097 - accuracy: 0.9973
216/1688 [==>...........................] - ETA: 7s - loss: 0.0098 - accuracy: 0.9970
227/1688 [===>..........................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9970
238/1688 [===>..........................] - ETA: 7s - loss: 0.0100 - accuracy: 0.9971
248/1688 [===>..........................] - ETA: 7s - loss: 0.0100 - accuracy: 0.9970
259/1688 [===>..........................] - ETA: 7s - loss: 0.0098 - accuracy: 0.9970
270/1688 [===>..........................] - ETA: 7s - loss: 0.0098 - accuracy: 0.9970
281/1688 [===>..........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9970
292/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9970
303/1688 [====>.........................] - ETA: 6s - loss: 0.0107 - accuracy: 0.9967
314/1688 [====>.........................] - ETA: 6s - loss: 0.0106 - accuracy: 0.9966
325/1688 [====>.........................] - ETA: 6s - loss: 0.0105 - accuracy: 0.9966
336/1688 [====>.........................] - ETA: 6s - loss: 0.0104 - accuracy: 0.9967
347/1688 [=====>........................] - ETA: 6s - loss: 0.0104 - accuracy: 0.9967
358/1688 [=====>........................] - ETA: 6s - loss: 0.0106 - accuracy: 0.9966
369/1688 [=====>........................] - ETA: 6s - loss: 0.0105 - accuracy: 0.9966
380/1688 [=====>........................] - ETA: 6s - loss: 0.0106 - accuracy: 0.9965
391/1688 [=====>........................] - ETA: 6s - loss: 0.0106 - accuracy: 0.9966
402/1688 [======>.......................] - ETA: 6s - loss: 0.0105 - accuracy: 0.9966
413/1688 [======>.......................] - ETA: 6s - loss: 0.0104 - accuracy: 0.9967
424/1688 [======>.......................] - ETA: 6s - loss: 0.0105 - accuracy: 0.9967
434/1688 [======>.......................] - ETA: 6s - loss: 0.0105 - accuracy: 0.9967
445/1688 [======>.......................] - ETA: 6s - loss: 0.0106 - accuracy: 0.9967
456/1688 [=======>......................] - ETA: 6s - loss: 0.0106 - accuracy: 0.9966
467/1688 [=======>......................] - ETA: 6s - loss: 0.0107 - accuracy: 0.9965
477/1688 [=======>......................] - ETA: 5s - loss: 0.0108 - accuracy: 0.9965
488/1688 [=======>......................] - ETA: 5s - loss: 0.0106 - accuracy: 0.9965
499/1688 [=======>......................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9966
510/1688 [========>.....................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9966
521/1688 [========>.....................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9966
532/1688 [========>.....................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9967
543/1688 [========>.....................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9967
554/1688 [========>.....................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9967
565/1688 [=========>....................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9968
575/1688 [=========>....................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9967
586/1688 [=========>....................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9968
597/1688 [=========>....................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9966
608/1688 [=========>....................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9966
618/1688 [=========>....................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9966
628/1688 [==========>...................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9966
639/1688 [==========>...................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9967
650/1688 [==========>...................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9967
660/1688 [==========>...................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9967
670/1688 [==========>...................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9968
681/1688 [===========>..................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9968
692/1688 [===========>..................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9968
703/1688 [===========>..................] - ETA: 4s - loss: 0.0100 - accuracy: 0.9969
714/1688 [===========>..................] - ETA: 4s - loss: 0.0100 - accuracy: 0.9969
725/1688 [===========>..................] - ETA: 4s - loss: 0.0099 - accuracy: 0.9969
736/1688 [============>.................] - ETA: 4s - loss: 0.0099 - accuracy: 0.9969
747/1688 [============>.................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9970
758/1688 [============>.................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9970
769/1688 [============>.................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9971
780/1688 [============>.................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9971
791/1688 [=============>................] - ETA: 4s - loss: 0.0095 - accuracy: 0.9971
802/1688 [=============>................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9971
812/1688 [=============>................] - ETA: 4s - loss: 0.0095 - accuracy: 0.9971
822/1688 [=============>................] - ETA: 4s - loss: 0.0095 - accuracy: 0.9971
833/1688 [=============>................] - ETA: 4s - loss: 0.0095 - accuracy: 0.9971
844/1688 [==============>...............] - ETA: 4s - loss: 0.0094 - accuracy: 0.9972
855/1688 [==============>...............] - ETA: 4s - loss: 0.0094 - accuracy: 0.9972
866/1688 [==============>...............] - ETA: 4s - loss: 0.0093 - accuracy: 0.9973
877/1688 [==============>...............] - ETA: 4s - loss: 0.0092 - accuracy: 0.9973
887/1688 [==============>...............] - ETA: 3s - loss: 0.0093 - accuracy: 0.9972
897/1688 [==============>...............] - ETA: 3s - loss: 0.0093 - accuracy: 0.9972
908/1688 [===============>..............] - ETA: 3s - loss: 0.0093 - accuracy: 0.9972
918/1688 [===============>..............] - ETA: 3s - loss: 0.0092 - accuracy: 0.9972
928/1688 [===============>..............] - ETA: 3s - loss: 0.0092 - accuracy: 0.9972
939/1688 [===============>..............] - ETA: 3s - loss: 0.0092 - accuracy: 0.9972
950/1688 [===============>..............] - ETA: 3s - loss: 0.0093 - accuracy: 0.9972
961/1688 [================>.............] - ETA: 3s - loss: 0.0093 - accuracy: 0.9972
972/1688 [================>.............] - ETA: 3s - loss: 0.0093 - accuracy: 0.9972
983/1688 [================>.............] - ETA: 3s - loss: 0.0092 - accuracy: 0.9972
994/1688 [================>.............] - ETA: 3s - loss: 0.0092 - accuracy: 0.9972
1005/1688 [================>.............] - ETA: 3s - loss: 0.0092 - accuracy: 0.9973
1016/1688 [=================>............] - ETA: 3s - loss: 0.0092 - accuracy: 0.9973
1026/1688 [=================>............] - ETA: 3s - loss: 0.0091 - accuracy: 0.9973
1037/1688 [=================>............] - ETA: 3s - loss: 0.0091 - accuracy: 0.9973
1048/1688 [=================>............] - ETA: 3s - loss: 0.0091 - accuracy: 0.9973
1058/1688 [=================>............] - ETA: 3s - loss: 0.0091 - accuracy: 0.9973
1068/1688 [=================>............] - ETA: 3s - loss: 0.0090 - accuracy: 0.9973
1079/1688 [==================>...........] - ETA: 3s - loss: 0.0090 - accuracy: 0.9973
1089/1688 [==================>...........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1100/1688 [==================>...........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1111/1688 [==================>...........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1122/1688 [==================>...........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1133/1688 [===================>..........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1143/1688 [===================>..........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9973
1154/1688 [===================>..........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1165/1688 [===================>..........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1186/1688 [====================>.........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1197/1688 [====================>.........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9974
1208/1688 [====================>.........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9974
1219/1688 [====================>.........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9973
1230/1688 [====================>.........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9974
1241/1688 [=====================>........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9973
1252/1688 [=====================>........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9974
1263/1688 [=====================>........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9974
1273/1688 [=====================>........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9974
1284/1688 [=====================>........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9974
1294/1688 [=====================>........] - ETA: 1s - loss: 0.0088 - accuracy: 0.9974
1305/1688 [======================>.......] - ETA: 1s - loss: 0.0088 - accuracy: 0.9974
1316/1688 [======================>.......] - ETA: 1s - loss: 0.0088 - accuracy: 0.9974
1326/1688 [======================>.......] - ETA: 1s - loss: 0.0087 - accuracy: 0.9975
1336/1688 [======================>.......] - ETA: 1s - loss: 0.0087 - accuracy: 0.9975
1347/1688 [======================>.......] - ETA: 1s - loss: 0.0087 - accuracy: 0.9974
1357/1688 [=======================>......] - ETA: 1s - loss: 0.0086 - accuracy: 0.9975
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0087 - accuracy: 0.9975
1379/1688 [=======================>......] - ETA: 1s - loss: 0.0087 - accuracy: 0.9975
1390/1688 [=======================>......] - ETA: 1s - loss: 0.0088 - accuracy: 0.9975
1401/1688 [=======================>......] - ETA: 1s - loss: 0.0087 - accuracy: 0.9975
1411/1688 [========================>.....] - ETA: 1s - loss: 0.0087 - accuracy: 0.9975
1422/1688 [========================>.....] - ETA: 1s - loss: 0.0088 - accuracy: 0.9974
1432/1688 [========================>.....] - ETA: 1s - loss: 0.0087 - accuracy: 0.9974
1443/1688 [========================>.....] - ETA: 1s - loss: 0.0087 - accuracy: 0.9975
1454/1688 [========================>.....] - ETA: 1s - loss: 0.0087 - accuracy: 0.9975
1465/1688 [=========================>....] - ETA: 1s - loss: 0.0087 - accuracy: 0.9974
1476/1688 [=========================>....] - ETA: 1s - loss: 0.0087 - accuracy: 0.9975
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0086 - accuracy: 0.9975
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0086 - accuracy: 0.9975
1507/1688 [=========================>....] - ETA: 0s - loss: 0.0086 - accuracy: 0.9975
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0085 - accuracy: 0.9975
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0085 - accuracy: 0.9975
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0085 - accuracy: 0.9975
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0085 - accuracy: 0.9975
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0085 - accuracy: 0.9975
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0085 - accuracy: 0.9975
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0085 - accuracy: 0.9975
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0084 - accuracy: 0.9975
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0084 - accuracy: 0.9975
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0084 - accuracy: 0.9975
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0084 - accuracy: 0.9975
1636/1688 [============================>.] - ETA: 0s - loss: 0.0084 - accuracy: 0.9975
1647/1688 [============================>.] - ETA: 0s - loss: 0.0084 - accuracy: 0.9975
1658/1688 [============================>.] - ETA: 0s - loss: 0.0084 - accuracy: 0.9975
1669/1688 [============================>.] - ETA: 0s - loss: 0.0084 - accuracy: 0.9975
1680/1688 [============================>.] - ETA: 0s - loss: 0.0084 - accuracy: 0.9975
1688/1688 [==============================] - 15s 6ms/step - loss: 0.0084 - accuracy: 0.9975 - val_loss: 0.0487 - val_accuracy: 0.9888
Epoch 2/5
1/1688 [..............................] - ETA: 8s - loss: 1.4566e-04 - accuracy: 1.0000
12/1688 [..............................] - ETA: 8s - loss: 0.0025 - accuracy: 1.0000
23/1688 [..............................] - ETA: 8s - loss: 0.0043 - accuracy: 0.9986
34/1688 [..............................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9982
45/1688 [..............................] - ETA: 8s - loss: 0.0058 - accuracy: 0.9986
55/1688 [..............................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9989
66/1688 [>.............................] - ETA: 8s - loss: 0.0055 - accuracy: 0.9986
77/1688 [>.............................] - ETA: 7s - loss: 0.0050 - accuracy: 0.9988
87/1688 [>.............................] - ETA: 7s - loss: 0.0049 - accuracy: 0.9986
98/1688 [>.............................] - ETA: 7s - loss: 0.0047 - accuracy: 0.9987
109/1688 [>.............................] - ETA: 7s - loss: 0.0048 - accuracy: 0.9989
120/1688 [=>............................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9990
130/1688 [=>............................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9986
140/1688 [=>............................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9987
151/1688 [=>............................] - ETA: 7s - loss: 0.0047 - accuracy: 0.9986
162/1688 [=>............................] - ETA: 7s - loss: 0.0047 - accuracy: 0.9986
173/1688 [==>...........................] - ETA: 7s - loss: 0.0048 - accuracy: 0.9986
183/1688 [==>...........................] - ETA: 7s - loss: 0.0047 - accuracy: 0.9986
194/1688 [==>...........................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9986
205/1688 [==>...........................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9986
216/1688 [==>...........................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9987
226/1688 [===>..........................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9988
237/1688 [===>..........................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9988
248/1688 [===>..........................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9989
259/1688 [===>..........................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9989
270/1688 [===>..........................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9990
281/1688 [===>..........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9990
292/1688 [====>.........................] - ETA: 6s - loss: 0.0053 - accuracy: 0.9988
302/1688 [====>.........................] - ETA: 6s - loss: 0.0053 - accuracy: 0.9989
313/1688 [====>.........................] - ETA: 6s - loss: 0.0052 - accuracy: 0.9989
324/1688 [====>.........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
335/1688 [====>.........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9990
346/1688 [=====>........................] - ETA: 6s - loss: 0.0052 - accuracy: 0.9988
357/1688 [=====>........................] - ETA: 6s - loss: 0.0052 - accuracy: 0.9989
368/1688 [=====>........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
379/1688 [=====>........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
390/1688 [=====>........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9990
400/1688 [======>.......................] - ETA: 6s - loss: 0.0052 - accuracy: 0.9989
411/1688 [======>.......................] - ETA: 6s - loss: 0.0053 - accuracy: 0.9989
422/1688 [======>.......................] - ETA: 6s - loss: 0.0053 - accuracy: 0.9989
433/1688 [======>.......................] - ETA: 6s - loss: 0.0056 - accuracy: 0.9988
444/1688 [======>.......................] - ETA: 6s - loss: 0.0056 - accuracy: 0.9987
455/1688 [=======>......................] - ETA: 6s - loss: 0.0055 - accuracy: 0.9988
465/1688 [=======>......................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9987
476/1688 [=======>......................] - ETA: 6s - loss: 0.0058 - accuracy: 0.9988
487/1688 [=======>......................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9988
498/1688 [=======>......................] - ETA: 5s - loss: 0.0057 - accuracy: 0.9988
508/1688 [========>.....................] - ETA: 5s - loss: 0.0056 - accuracy: 0.9988
519/1688 [========>.....................] - ETA: 5s - loss: 0.0057 - accuracy: 0.9989
530/1688 [========>.....................] - ETA: 5s - loss: 0.0056 - accuracy: 0.9989
540/1688 [========>.....................] - ETA: 5s - loss: 0.0057 - accuracy: 0.9988
551/1688 [========>.....................] - ETA: 5s - loss: 0.0056 - accuracy: 0.9989
562/1688 [========>.....................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
573/1688 [=========>....................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
584/1688 [=========>....................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
594/1688 [=========>....................] - ETA: 5s - loss: 0.0054 - accuracy: 0.9989
605/1688 [=========>....................] - ETA: 5s - loss: 0.0054 - accuracy: 0.9990
616/1688 [=========>....................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
627/1688 [==========>...................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
637/1688 [==========>...................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
648/1688 [==========>...................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
659/1688 [==========>...................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
669/1688 [==========>...................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
680/1688 [===========>..................] - ETA: 5s - loss: 0.0055 - accuracy: 0.9989
690/1688 [===========>..................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
701/1688 [===========>..................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
712/1688 [===========>..................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
723/1688 [===========>..................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
734/1688 [============>.................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
745/1688 [============>.................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
756/1688 [============>.................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
766/1688 [============>.................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
777/1688 [============>.................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
788/1688 [=============>................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
799/1688 [=============>................] - ETA: 4s - loss: 0.0055 - accuracy: 0.9989
810/1688 [=============>................] - ETA: 4s - loss: 0.0054 - accuracy: 0.9990
821/1688 [=============>................] - ETA: 4s - loss: 0.0054 - accuracy: 0.9990
832/1688 [=============>................] - ETA: 4s - loss: 0.0054 - accuracy: 0.9989
843/1688 [=============>................] - ETA: 4s - loss: 0.0054 - accuracy: 0.9990
854/1688 [==============>...............] - ETA: 4s - loss: 0.0053 - accuracy: 0.9990
865/1688 [==============>...............] - ETA: 4s - loss: 0.0053 - accuracy: 0.9990
876/1688 [==============>...............] - ETA: 4s - loss: 0.0053 - accuracy: 0.9990
887/1688 [==============>...............] - ETA: 3s - loss: 0.0053 - accuracy: 0.9990
897/1688 [==============>...............] - ETA: 3s - loss: 0.0053 - accuracy: 0.9990
908/1688 [===============>..............] - ETA: 3s - loss: 0.0053 - accuracy: 0.9990
919/1688 [===============>..............] - ETA: 3s - loss: 0.0053 - accuracy: 0.9989
930/1688 [===============>..............] - ETA: 3s - loss: 0.0054 - accuracy: 0.9989
940/1688 [===============>..............] - ETA: 3s - loss: 0.0053 - accuracy: 0.9989
951/1688 [===============>..............] - ETA: 3s - loss: 0.0054 - accuracy: 0.9989
962/1688 [================>.............] - ETA: 3s - loss: 0.0053 - accuracy: 0.9990
973/1688 [================>.............] - ETA: 3s - loss: 0.0054 - accuracy: 0.9989
984/1688 [================>.............] - ETA: 3s - loss: 0.0053 - accuracy: 0.9989
995/1688 [================>.............] - ETA: 3s - loss: 0.0054 - accuracy: 0.9989
1006/1688 [================>.............] - ETA: 3s - loss: 0.0055 - accuracy: 0.9989
1017/1688 [=================>............] - ETA: 3s - loss: 0.0055 - accuracy: 0.9989
1028/1688 [=================>............] - ETA: 3s - loss: 0.0054 - accuracy: 0.9989
1039/1688 [=================>............] - ETA: 3s - loss: 0.0055 - accuracy: 0.9989
1050/1688 [=================>............] - ETA: 3s - loss: 0.0055 - accuracy: 0.9988
1061/1688 [=================>............] - ETA: 3s - loss: 0.0057 - accuracy: 0.9988
1072/1688 [==================>...........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9988
1083/1688 [==================>...........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9988
1094/1688 [==================>...........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9988
1105/1688 [==================>...........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9988
1116/1688 [==================>...........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9988
1127/1688 [===================>..........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9988
1138/1688 [===================>..........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9988
1149/1688 [===================>..........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9987
1159/1688 [===================>..........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9988
1170/1688 [===================>..........] - ETA: 2s - loss: 0.0056 - accuracy: 0.9988
1181/1688 [===================>..........] - ETA: 2s - loss: 0.0056 - accuracy: 0.9988
1192/1688 [====================>.........] - ETA: 2s - loss: 0.0056 - accuracy: 0.9988
1202/1688 [====================>.........] - ETA: 2s - loss: 0.0056 - accuracy: 0.9988
1213/1688 [====================>.........] - ETA: 2s - loss: 0.0056 - accuracy: 0.9988
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0056 - accuracy: 0.9988
1235/1688 [====================>.........] - ETA: 2s - loss: 0.0056 - accuracy: 0.9987
1246/1688 [=====================>........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9987
1257/1688 [=====================>........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9987
1268/1688 [=====================>........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9986
1278/1688 [=====================>........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9987
1289/1688 [=====================>........] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1299/1688 [======================>.......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9987
1310/1688 [======================>.......] - ETA: 1s - loss: 0.0056 - accuracy: 0.9987
1321/1688 [======================>.......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1343/1688 [======================>.......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1354/1688 [=======================>......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1365/1688 [=======================>......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1375/1688 [=======================>......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1386/1688 [=======================>......] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1397/1688 [=======================>......] - ETA: 1s - loss: 0.0056 - accuracy: 0.9986
1408/1688 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1419/1688 [========================>.....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9986
1429/1688 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1439/1688 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1449/1688 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1460/1688 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1471/1688 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1481/1688 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9986
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0058 - accuracy: 0.9985
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9985
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9985
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9985
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9985
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9985
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1601/1688 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1634/1688 [============================>.] - ETA: 0s - loss: 0.0058 - accuracy: 0.9985
1645/1688 [============================>.] - ETA: 0s - loss: 0.0058 - accuracy: 0.9985
1655/1688 [============================>.] - ETA: 0s - loss: 0.0058 - accuracy: 0.9985
1666/1688 [============================>.] - ETA: 0s - loss: 0.0058 - accuracy: 0.9986
1677/1688 [============================>.] - ETA: 0s - loss: 0.0058 - accuracy: 0.9986
1688/1688 [==============================] - ETA: 0s - loss: 0.0058 - accuracy: 0.9986
1688/1688 [==============================] - 9s 5ms/step - loss: 0.0058 - accuracy: 0.9986 - val_loss: 0.0504 - val_accuracy: 0.9877
Epoch 3/5
1/1688 [..............................] - ETA: 8s - loss: 0.0019 - accuracy: 1.0000
12/1688 [..............................] - ETA: 8s - loss: 0.0034 - accuracy: 1.0000
23/1688 [..............................] - ETA: 8s - loss: 0.0036 - accuracy: 1.0000
34/1688 [..............................] - ETA: 8s - loss: 0.0032 - accuracy: 1.0000
45/1688 [..............................] - ETA: 8s - loss: 0.0043 - accuracy: 0.9986
55/1688 [..............................] - ETA: 8s - loss: 0.0037 - accuracy: 0.9989
66/1688 [>.............................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9991
77/1688 [>.............................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9988
88/1688 [>.............................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9989
98/1688 [>.............................] - ETA: 7s - loss: 0.0041 - accuracy: 0.9990
108/1688 [>.............................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9991
119/1688 [=>............................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9989
129/1688 [=>............................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9990
139/1688 [=>............................] - ETA: 7s - loss: 0.0048 - accuracy: 0.9991
150/1688 [=>............................] - ETA: 7s - loss: 0.0049 - accuracy: 0.9990
161/1688 [=>............................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9988
172/1688 [==>...........................] - ETA: 7s - loss: 0.0049 - accuracy: 0.9989
183/1688 [==>...........................] - ETA: 7s - loss: 0.0048 - accuracy: 0.9988
194/1688 [==>...........................] - ETA: 7s - loss: 0.0047 - accuracy: 0.9989
205/1688 [==>...........................] - ETA: 7s - loss: 0.0047 - accuracy: 0.9988
216/1688 [==>...........................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9988
227/1688 [===>..........................] - ETA: 7s - loss: 0.0049 - accuracy: 0.9986
238/1688 [===>..........................] - ETA: 7s - loss: 0.0056 - accuracy: 0.9984
249/1688 [===>..........................] - ETA: 7s - loss: 0.0055 - accuracy: 0.9985
259/1688 [===>..........................] - ETA: 7s - loss: 0.0056 - accuracy: 0.9984
270/1688 [===>..........................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9985
281/1688 [===>..........................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9986
292/1688 [====>.........................] - ETA: 6s - loss: 0.0053 - accuracy: 0.9986
303/1688 [====>.........................] - ETA: 6s - loss: 0.0055 - accuracy: 0.9985
313/1688 [====>.........................] - ETA: 6s - loss: 0.0055 - accuracy: 0.9984
324/1688 [====>.........................] - ETA: 6s - loss: 0.0056 - accuracy: 0.9984
335/1688 [====>.........................] - ETA: 6s - loss: 0.0058 - accuracy: 0.9983
346/1688 [=====>........................] - ETA: 6s - loss: 0.0058 - accuracy: 0.9983
357/1688 [=====>........................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9983
367/1688 [=====>........................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9983
378/1688 [=====>........................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9983
389/1688 [=====>........................] - ETA: 6s - loss: 0.0058 - accuracy: 0.9984
400/1688 [======>.......................] - ETA: 6s - loss: 0.0058 - accuracy: 0.9984
411/1688 [======>.......................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9985
422/1688 [======>.......................] - ETA: 6s - loss: 0.0056 - accuracy: 0.9985
432/1688 [======>.......................] - ETA: 6s - loss: 0.0055 - accuracy: 0.9986
442/1688 [======>.......................] - ETA: 6s - loss: 0.0055 - accuracy: 0.9986
453/1688 [=======>......................] - ETA: 6s - loss: 0.0055 - accuracy: 0.9986
464/1688 [=======>......................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9987
475/1688 [=======>......................] - ETA: 6s - loss: 0.0053 - accuracy: 0.9987
486/1688 [=======>......................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
497/1688 [=======>......................] - ETA: 5s - loss: 0.0053 - accuracy: 0.9987
508/1688 [========>.....................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
519/1688 [========>.....................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9987
530/1688 [========>.....................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
541/1688 [========>.....................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9987
552/1688 [========>.....................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
563/1688 [=========>....................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
573/1688 [=========>....................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
584/1688 [=========>....................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9987
595/1688 [=========>....................] - ETA: 5s - loss: 0.0053 - accuracy: 0.9987
606/1688 [=========>....................] - ETA: 5s - loss: 0.0053 - accuracy: 0.9987
617/1688 [=========>....................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
628/1688 [==========>...................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
638/1688 [==========>...................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
649/1688 [==========>...................] - ETA: 5s - loss: 0.0052 - accuracy: 0.9987
659/1688 [==========>...................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9987
670/1688 [==========>...................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9987
681/1688 [===========>..................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9988
692/1688 [===========>..................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9988
703/1688 [===========>..................] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
714/1688 [===========>..................] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
725/1688 [===========>..................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
736/1688 [============>.................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
747/1688 [============>.................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9987
758/1688 [============>.................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
769/1688 [============>.................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9987
779/1688 [============>.................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9987
789/1688 [=============>................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
800/1688 [=============>................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
810/1688 [=============>................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9988
821/1688 [=============>................] - ETA: 4s - loss: 0.0051 - accuracy: 0.9988
832/1688 [=============>................] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
843/1688 [=============>................] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
854/1688 [==============>...............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
865/1688 [==============>...............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
875/1688 [==============>...............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
886/1688 [==============>...............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9988
897/1688 [==============>...............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9987
907/1688 [===============>..............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9988
918/1688 [===============>..............] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
929/1688 [===============>..............] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
940/1688 [===============>..............] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
950/1688 [===============>..............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
960/1688 [================>.............] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
970/1688 [================>.............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
980/1688 [================>.............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
990/1688 [================>.............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
1001/1688 [================>.............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
1012/1688 [================>.............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
1023/1688 [=================>............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
1034/1688 [=================>............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9989
1045/1688 [=================>............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
1055/1688 [=================>............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
1065/1688 [=================>............] - ETA: 3s - loss: 0.0048 - accuracy: 0.9989
1075/1688 [==================>...........] - ETA: 3s - loss: 0.0048 - accuracy: 0.9988
1086/1688 [==================>...........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9988
1096/1688 [==================>...........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1106/1688 [==================>...........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1116/1688 [==================>...........] - ETA: 2s - loss: 0.0047 - accuracy: 0.9989
1127/1688 [===================>..........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1138/1688 [===================>..........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1148/1688 [===================>..........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1159/1688 [===================>..........] - ETA: 2s - loss: 0.0047 - accuracy: 0.9989
1169/1688 [===================>..........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1180/1688 [===================>..........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1191/1688 [====================>.........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1202/1688 [====================>.........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1223/1688 [====================>.........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1234/1688 [====================>.........] - ETA: 2s - loss: 0.0048 - accuracy: 0.9989
1244/1688 [=====================>........] - ETA: 2s - loss: 0.0047 - accuracy: 0.9989
1255/1688 [=====================>........] - ETA: 2s - loss: 0.0047 - accuracy: 0.9989
1265/1688 [=====================>........] - ETA: 2s - loss: 0.0047 - accuracy: 0.9989
1276/1688 [=====================>........] - ETA: 2s - loss: 0.0047 - accuracy: 0.9989
1287/1688 [=====================>........] - ETA: 1s - loss: 0.0047 - accuracy: 0.9989
1298/1688 [======================>.......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9989
1308/1688 [======================>.......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9989
1318/1688 [======================>.......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9989
1328/1688 [======================>.......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9989
1339/1688 [======================>.......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9989
1349/1688 [======================>.......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9989
1360/1688 [=======================>......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9989
1371/1688 [=======================>......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9990
1381/1688 [=======================>......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9990
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9989
1403/1688 [=======================>......] - ETA: 1s - loss: 0.0047 - accuracy: 0.9990
1414/1688 [========================>.....] - ETA: 1s - loss: 0.0047 - accuracy: 0.9990
1424/1688 [========================>.....] - ETA: 1s - loss: 0.0047 - accuracy: 0.9990
1434/1688 [========================>.....] - ETA: 1s - loss: 0.0046 - accuracy: 0.9990
1445/1688 [========================>.....] - ETA: 1s - loss: 0.0046 - accuracy: 0.9990
1456/1688 [========================>.....] - ETA: 1s - loss: 0.0046 - accuracy: 0.9990
1466/1688 [=========================>....] - ETA: 1s - loss: 0.0046 - accuracy: 0.9990
1477/1688 [=========================>....] - ETA: 1s - loss: 0.0046 - accuracy: 0.9990
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0047 - accuracy: 0.9990
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0047 - accuracy: 0.9990
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0047 - accuracy: 0.9990
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0047 - accuracy: 0.9990
1529/1688 [==========================>...] - ETA: 0s - loss: 0.0047 - accuracy: 0.9990
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1636/1688 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9990
1647/1688 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9990
1658/1688 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9990
1668/1688 [============================>.] - ETA: 0s - loss: 0.0046 - accuracy: 0.9990
1679/1688 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9990
1688/1688 [==============================] - 9s 5ms/step - loss: 0.0045 - accuracy: 0.9990 - val_loss: 0.0490 - val_accuracy: 0.9890
Epoch 4/5
1/1688 [..............................] - ETA: 8s - loss: 0.0058 - accuracy: 1.0000
12/1688 [..............................] - ETA: 8s - loss: 0.0041 - accuracy: 1.0000
22/1688 [..............................] - ETA: 8s - loss: 0.0032 - accuracy: 1.0000
33/1688 [..............................] - ETA: 8s - loss: 0.0024 - accuracy: 1.0000
44/1688 [..............................] - ETA: 8s - loss: 0.0025 - accuracy: 1.0000
55/1688 [..............................] - ETA: 8s - loss: 0.0025 - accuracy: 1.0000
66/1688 [>.............................] - ETA: 8s - loss: 0.0022 - accuracy: 1.0000
77/1688 [>.............................] - ETA: 7s - loss: 0.0025 - accuracy: 0.9996
88/1688 [>.............................] - ETA: 7s - loss: 0.0025 - accuracy: 0.9996
98/1688 [>.............................] - ETA: 7s - loss: 0.0029 - accuracy: 0.9994
108/1688 [>.............................] - ETA: 7s - loss: 0.0029 - accuracy: 0.9994
119/1688 [=>............................] - ETA: 7s - loss: 0.0029 - accuracy: 0.9995
129/1688 [=>............................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9995
140/1688 [=>............................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9996
150/1688 [=>............................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9996
161/1688 [=>............................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9996
172/1688 [==>...........................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9996
183/1688 [==>...........................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9997
194/1688 [==>...........................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9997
205/1688 [==>...........................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9997
215/1688 [==>...........................] - ETA: 7s - loss: 0.0035 - accuracy: 0.9996
225/1688 [==>...........................] - ETA: 7s - loss: 0.0034 - accuracy: 0.9996
235/1688 [===>..........................] - ETA: 7s - loss: 0.0035 - accuracy: 0.9995
246/1688 [===>..........................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9995
257/1688 [===>..........................] - ETA: 7s - loss: 0.0035 - accuracy: 0.9995
268/1688 [===>..........................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9994
279/1688 [===>..........................] - ETA: 7s - loss: 0.0035 - accuracy: 0.9994
290/1688 [====>.........................] - ETA: 6s - loss: 0.0037 - accuracy: 0.9994
300/1688 [====>.........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9994
310/1688 [====>.........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9994
320/1688 [====>.........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9994
331/1688 [====>.........................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9994
342/1688 [=====>........................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9995
353/1688 [=====>........................] - ETA: 6s - loss: 0.0034 - accuracy: 0.9995
364/1688 [=====>........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9994
375/1688 [=====>........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9993
386/1688 [=====>........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9994
397/1688 [======>.......................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9994
407/1688 [======>.......................] - ETA: 6s - loss: 0.0037 - accuracy: 0.9994
418/1688 [======>.......................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9993
429/1688 [======>.......................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9993
439/1688 [======>.......................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9994
450/1688 [======>.......................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9992
461/1688 [=======>......................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9993
471/1688 [=======>......................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9993
482/1688 [=======>......................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9992
493/1688 [=======>......................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9992
503/1688 [=======>......................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9992
513/1688 [========>.....................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9992
524/1688 [========>.....................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9992
535/1688 [========>.....................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9991
546/1688 [========>.....................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9991
557/1688 [========>.....................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
568/1688 [=========>....................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
579/1688 [=========>....................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
590/1688 [=========>....................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9992
601/1688 [=========>....................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9992
612/1688 [=========>....................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
623/1688 [==========>...................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
634/1688 [==========>...................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9992
645/1688 [==========>...................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9992
656/1688 [==========>...................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9992
667/1688 [==========>...................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9992
678/1688 [===========>..................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9992
689/1688 [===========>..................] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
699/1688 [===========>..................] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
710/1688 [===========>..................] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
721/1688 [===========>..................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9992
732/1688 [============>.................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9991
742/1688 [============>.................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9992
753/1688 [============>.................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9991
764/1688 [============>.................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9991
775/1688 [============>.................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9992
785/1688 [============>.................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9992
796/1688 [=============>................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9992
807/1688 [=============>................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9991
818/1688 [=============>................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9992
829/1688 [=============>................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9992
839/1688 [=============>................] - ETA: 4s - loss: 0.0039 - accuracy: 0.9992
850/1688 [==============>...............] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
861/1688 [==============>...............] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
872/1688 [==============>...............] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
883/1688 [==============>...............] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
894/1688 [==============>...............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9992
905/1688 [===============>..............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9992
915/1688 [===============>..............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9992
925/1688 [===============>..............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9992
936/1688 [===============>..............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9992
946/1688 [===============>..............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9992
957/1688 [================>.............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9992
968/1688 [================>.............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
979/1688 [================>.............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
989/1688 [================>.............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
999/1688 [================>.............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
1010/1688 [================>.............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
1021/1688 [=================>............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
1032/1688 [=================>............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9992
1043/1688 [=================>............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
1054/1688 [=================>............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
1065/1688 [=================>............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
1076/1688 [==================>...........] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
1087/1688 [==================>...........] - ETA: 2s - loss: 0.0037 - accuracy: 0.9993
1097/1688 [==================>...........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1107/1688 [==================>...........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1117/1688 [==================>...........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9992
1128/1688 [===================>..........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9993
1139/1688 [===================>..........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1150/1688 [===================>..........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1161/1688 [===================>..........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9992
1172/1688 [===================>..........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9993
1183/1688 [====================>.........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9992
1194/1688 [====================>.........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9992
1205/1688 [====================>.........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9992
1216/1688 [====================>.........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1227/1688 [====================>.........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1238/1688 [=====================>........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1249/1688 [=====================>........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9992
1260/1688 [=====================>........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9992
1271/1688 [=====================>........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9992
1282/1688 [=====================>........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9992
1292/1688 [=====================>........] - ETA: 1s - loss: 0.0039 - accuracy: 0.9992
1303/1688 [======================>.......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9992
1314/1688 [======================>.......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9992
1325/1688 [======================>.......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9992
1336/1688 [======================>.......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9992
1347/1688 [======================>.......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1357/1688 [=======================>......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1378/1688 [=======================>......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1389/1688 [=======================>......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1400/1688 [=======================>......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1410/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1421/1688 [========================>.....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1431/1688 [========================>.....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1442/1688 [========================>.....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1453/1688 [========================>.....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1463/1688 [=========================>....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1474/1688 [=========================>....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1484/1688 [=========================>....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1506/1688 [=========================>....] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1538/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1592/1688 [===========================>..] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1603/1688 [===========================>..] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1634/1688 [============================>.] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1645/1688 [============================>.] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1656/1688 [============================>.] - ETA: 0s - loss: 0.0040 - accuracy: 0.9992
1667/1688 [============================>.] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1678/1688 [============================>.] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1688/1688 [==============================] - 9s 5ms/step - loss: 0.0040 - accuracy: 0.9991 - val_loss: 0.0484 - val_accuracy: 0.9888
Epoch 5/5
1/1688 [..............................] - ETA: 8s - loss: 3.5689e-04 - accuracy: 1.0000
12/1688 [..............................] - ETA: 8s - loss: 0.0101 - accuracy: 0.9974
23/1688 [..............................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9986
33/1688 [..............................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9991
44/1688 [..............................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9986
55/1688 [..............................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9989
66/1688 [>.............................] - ETA: 8s - loss: 0.0044 - accuracy: 0.9991
77/1688 [>.............................] - ETA: 7s - loss: 0.0044 - accuracy: 0.9992
88/1688 [>.............................] - ETA: 7s - loss: 0.0041 - accuracy: 0.9993
99/1688 [>.............................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9994
109/1688 [>.............................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9994
119/1688 [=>............................] - ETA: 7s - loss: 0.0035 - accuracy: 0.9995
130/1688 [=>............................] - ETA: 7s - loss: 0.0034 - accuracy: 0.9995
140/1688 [=>............................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9996
151/1688 [=>............................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9996
162/1688 [=>............................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9996
173/1688 [==>...........................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9996
184/1688 [==>...........................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9997
194/1688 [==>...........................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9997
205/1688 [==>...........................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9997
216/1688 [==>...........................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9997
227/1688 [===>..........................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9997
238/1688 [===>..........................] - ETA: 7s - loss: 0.0030 - accuracy: 0.9997
248/1688 [===>..........................] - ETA: 7s - loss: 0.0030 - accuracy: 0.9997
258/1688 [===>..........................] - ETA: 7s - loss: 0.0030 - accuracy: 0.9998
269/1688 [===>..........................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9997
279/1688 [===>..........................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9997
289/1688 [====>.........................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9997
300/1688 [====>.........................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9997
311/1688 [====>.........................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
322/1688 [====>.........................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
333/1688 [====>.........................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
344/1688 [=====>........................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
355/1688 [=====>........................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
366/1688 [=====>........................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9997
377/1688 [=====>........................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9998
388/1688 [=====>........................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9998
399/1688 [======>.......................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9998
410/1688 [======>.......................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9997
421/1688 [======>.......................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
431/1688 [======>.......................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
442/1688 [======>.......................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
453/1688 [=======>......................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
464/1688 [=======>......................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9997
474/1688 [=======>......................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9996
485/1688 [=======>......................] - ETA: 5s - loss: 0.0031 - accuracy: 0.9996
495/1688 [=======>......................] - ETA: 5s - loss: 0.0030 - accuracy: 0.9996
505/1688 [=======>......................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
516/1688 [========>.....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
526/1688 [========>.....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
536/1688 [========>.....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
547/1688 [========>.....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
558/1688 [========>.....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
569/1688 [=========>....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
580/1688 [=========>....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
591/1688 [=========>....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
601/1688 [=========>....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
612/1688 [=========>....................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
623/1688 [==========>...................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
634/1688 [==========>...................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9997
645/1688 [==========>...................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
656/1688 [==========>...................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
667/1688 [==========>...................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
677/1688 [===========>..................] - ETA: 5s - loss: 0.0032 - accuracy: 0.9996
687/1688 [===========>..................] - ETA: 4s - loss: 0.0031 - accuracy: 0.9996
698/1688 [===========>..................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
708/1688 [===========>..................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
719/1688 [===========>..................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
729/1688 [===========>..................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
740/1688 [============>.................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9995
750/1688 [============>.................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9995
761/1688 [============>.................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9995
772/1688 [============>.................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
783/1688 [============>.................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
794/1688 [=============>................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
805/1688 [=============>................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9995
815/1688 [=============>................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9995
826/1688 [=============>................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9995
836/1688 [=============>................] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
847/1688 [==============>...............] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
858/1688 [==============>...............] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
869/1688 [==============>...............] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
880/1688 [==============>...............] - ETA: 4s - loss: 0.0032 - accuracy: 0.9996
891/1688 [==============>...............] - ETA: 3s - loss: 0.0032 - accuracy: 0.9996
902/1688 [===============>..............] - ETA: 3s - loss: 0.0032 - accuracy: 0.9995
913/1688 [===============>..............] - ETA: 3s - loss: 0.0032 - accuracy: 0.9996
924/1688 [===============>..............] - ETA: 3s - loss: 0.0032 - accuracy: 0.9996
935/1688 [===============>..............] - ETA: 3s - loss: 0.0033 - accuracy: 0.9996
946/1688 [===============>..............] - ETA: 3s - loss: 0.0033 - accuracy: 0.9996
957/1688 [================>.............] - ETA: 3s - loss: 0.0033 - accuracy: 0.9996
968/1688 [================>.............] - ETA: 3s - loss: 0.0033 - accuracy: 0.9996
979/1688 [================>.............] - ETA: 3s - loss: 0.0033 - accuracy: 0.9996
990/1688 [================>.............] - ETA: 3s - loss: 0.0033 - accuracy: 0.9996
1000/1688 [================>.............] - ETA: 3s - loss: 0.0033 - accuracy: 0.9996
1010/1688 [================>.............] - ETA: 3s - loss: 0.0032 - accuracy: 0.9996
1021/1688 [=================>............] - ETA: 3s - loss: 0.0033 - accuracy: 0.9996
1032/1688 [=================>............] - ETA: 3s - loss: 0.0032 - accuracy: 0.9996
1042/1688 [=================>............] - ETA: 3s - loss: 0.0032 - accuracy: 0.9996
1053/1688 [=================>............] - ETA: 3s - loss: 0.0032 - accuracy: 0.9996
1064/1688 [=================>............] - ETA: 3s - loss: 0.0032 - accuracy: 0.9996
1075/1688 [==================>...........] - ETA: 3s - loss: 0.0032 - accuracy: 0.9996
1085/1688 [==================>...........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1096/1688 [==================>...........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1107/1688 [==================>...........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1118/1688 [==================>...........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1129/1688 [===================>..........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1140/1688 [===================>..........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1151/1688 [===================>..........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1162/1688 [===================>..........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1173/1688 [===================>..........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1184/1688 [====================>.........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1195/1688 [====================>.........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1206/1688 [====================>.........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1217/1688 [====================>.........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1228/1688 [====================>.........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1239/1688 [=====================>........] - ETA: 2s - loss: 0.0033 - accuracy: 0.9996
1250/1688 [=====================>........] - ETA: 2s - loss: 0.0033 - accuracy: 0.9996
1261/1688 [=====================>........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1272/1688 [=====================>........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1282/1688 [=====================>........] - ETA: 2s - loss: 0.0032 - accuracy: 0.9996
1292/1688 [=====================>........] - ETA: 1s - loss: 0.0032 - accuracy: 0.9996
1303/1688 [======================>.......] - ETA: 1s - loss: 0.0032 - accuracy: 0.9996
1313/1688 [======================>.......] - ETA: 1s - loss: 0.0032 - accuracy: 0.9996
1322/1688 [======================>.......] - ETA: 1s - loss: 0.0032 - accuracy: 0.9996
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0033 - accuracy: 0.9996
1343/1688 [======================>.......] - ETA: 1s - loss: 0.0033 - accuracy: 0.9996
1354/1688 [=======================>......] - ETA: 1s - loss: 0.0032 - accuracy: 0.9996
1365/1688 [=======================>......] - ETA: 1s - loss: 0.0033 - accuracy: 0.9996
1376/1688 [=======================>......] - ETA: 1s - loss: 0.0032 - accuracy: 0.9996
1387/1688 [=======================>......] - ETA: 1s - loss: 0.0032 - accuracy: 0.9996
1398/1688 [=======================>......] - ETA: 1s - loss: 0.0033 - accuracy: 0.9996
1409/1688 [========================>.....] - ETA: 1s - loss: 0.0033 - accuracy: 0.9996
1419/1688 [========================>.....] - ETA: 1s - loss: 0.0033 - accuracy: 0.9996
1430/1688 [========================>.....] - ETA: 1s - loss: 0.0033 - accuracy: 0.9996
1441/1688 [========================>.....] - ETA: 1s - loss: 0.0033 - accuracy: 0.9996
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0033 - accuracy: 0.9995
1462/1688 [========================>.....] - ETA: 1s - loss: 0.0033 - accuracy: 0.9995
1473/1688 [=========================>....] - ETA: 1s - loss: 0.0033 - accuracy: 0.9995
1483/1688 [=========================>....] - ETA: 1s - loss: 0.0033 - accuracy: 0.9995
1493/1688 [=========================>....] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1526/1688 [==========================>...] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0034 - accuracy: 0.9994
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0034 - accuracy: 0.9994
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0035 - accuracy: 0.9994
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0034 - accuracy: 0.9994
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0034 - accuracy: 0.9994
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0034 - accuracy: 0.9994
1634/1688 [============================>.] - ETA: 0s - loss: 0.0034 - accuracy: 0.9994
1645/1688 [============================>.] - ETA: 0s - loss: 0.0034 - accuracy: 0.9994
1656/1688 [============================>.] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1667/1688 [============================>.] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1678/1688 [============================>.] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1688/1688 [==============================] - ETA: 0s - loss: 0.0034 - accuracy: 0.9995
1688/1688 [==============================] - 9s 5ms/step - loss: 0.0034 - accuracy: 0.9995 - val_loss: 0.0502 - val_accuracy: 0.9885
<keras.src.callbacks.History object at 0x7f4b012bee50>
score = model_quantized.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after fine tuning:', score)
Test accuracy after fine tuning: 0.9883999824523926
3. Convert
3.1 Convert to Akida model
When the quantized model produces satisfactory performance, it can be converted to the native Akida format. The convert function returns a model in Akida format ready for inference.
As with Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized)
model_akida.summary()
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 5
______________________________________________
__________________________________________________________________
Layer (type) Output shape Kernel shape
=============== SW/conv2d-dequantizer_2 (Software) ===============
conv2d (InputConv2D) [13, 13, 32] (3, 3, 1, 32)
__________________________________________________________________
depthwise_conv2d (DepthwiseConv2D) [7, 7, 32] (3, 3, 32, 1)
__________________________________________________________________
conv2d_1 (Conv2D) [7, 7, 64] (1, 1, 32, 64)
__________________________________________________________________
dense (Dense1D) [1, 1, 10] (3136, 10)
__________________________________________________________________
dequantizer_2 (Dequantizer) [1, 1, 10] N/A
__________________________________________________________________
3.2. Check performance
accuracy = model_akida.evaluate(x_test, y_test)
print('Test accuracy after conversion:', accuracy)
# For non-regression purposes
assert accuracy > 0.96
Test accuracy after conversion: 0.9857000112533569
3.3 Show predictions for a single image
Display one of the test images, such as the first image in the dataset from above, to visualize the output of the model.
# Test a single example
sample_image = 0
image = x_test[sample_image]
outputs = model_akida.predict(image.reshape(1, 28, 28, 1))
print('Input Label: %i' % y_test[sample_image])
f, axarr = plt.subplots(1, 2)
axarr[0].imshow(x_test[sample_image].reshape((28, 28)), cmap=cm.Greys_r)
axarr[0].set_title('Class %d' % y_test[sample_image])
axarr[1].bar(range(10), outputs.squeeze())
axarr[1].set_xticks(range(10))
plt.show()
print(outputs.squeeze())
Input Label: 7
[-16.708143 -8.418717 -5.9181294 -2.5619938 -20.631516 -8.099142
-29.145748 13.105431 -8.830281 -0.8264546]
Consider the output from the model above. As is typical in backprop-trained models, the final layer is a Dense layer with one neuron for each of the 10 classes in the dataset. The goal of training is to maximize the response of the neuron corresponding to the label of each training sample while minimizing the responses of the other neurons.
In the bar chart above, you can see the outputs from all 10 neurons. It is easy to see that neuron 7 responds much more strongly than the others. The first sample is indeed a number 7.
Total running time of the script: (2 minutes 10.899 seconds)