Note
Go to the end to download the full example code
Global Akida workflow
Using the MNIST dataset, this example shows the definition and training of a keras floating point model, its quantization to 8-bit with the help of calibration, its quantization to 4-bit using QAT and its conversion to Akida. Notice that the performance of the original keras floating point model is maintained throughout the Akida flow. Please refer to the Akida user guide for further information.
Note
Please refer to the TensorFlow tf.keras.models module for model creation/import details and the TensorFlow Guide for TensorFlow usage.
The MNIST example below is light enough so that a GPU is not needed for training.
Global Akida workflow
1. Create and train
1.1. Load and reshape MNIST dataset
import numpy as np
import matplotlib.cm as cm
import matplotlib.pyplot as plt
from keras.datasets import mnist
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Add a channels dimension to the image sets as Akida expects 4-D inputs (corresponding to
# (num_samples, width, height, channels). Note: MNIST is a grayscale dataset and is unusual
# in this respect - most image data already includes a channel dimension, and this step will
# not be necessary.
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
# Display a few images from the test set
f, axarr = plt.subplots(1, 4)
for i in range(0, 4):
axarr[i].imshow(x_test[i].reshape((28, 28)), cmap=cm.Greys_r)
axarr[i].set_title('Class %d' % y_test[i])
plt.show()
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
8192/11490434 [..............................] - ETA: 0s
24576/11490434 [..............................] - ETA: 1:16
40960/11490434 [..............................] - ETA: 1:03
57344/11490434 [..............................] - ETA: 1:01
81920/11490434 [..............................] - ETA: 52s
106496/11490434 [..............................] - ETA: 46s
139264/11490434 [..............................] - ETA: 40s
180224/11490434 [..............................] - ETA: 34s
229376/11490434 [..............................] - ETA: 29s
286720/11490434 [..............................] - ETA: 25s
360448/11490434 [..............................] - ETA: 22s
434176/11490434 [>.............................] - ETA: 19s
573440/11490434 [>.............................] - ETA: 15s
712704/11490434 [>.............................] - ETA: 13s
868352/11490434 [=>............................] - ETA: 11s
1081344/11490434 [=>............................] - ETA: 9s
1376256/11490434 [==>...........................] - ETA: 7s
1736704/11490434 [===>..........................] - ETA: 6s
2170880/11490434 [====>.........................] - ETA: 4s
2703360/11490434 [======>.......................] - ETA: 3s
3293184/11490434 [=======>......................] - ETA: 3s
3883008/11490434 [=========>....................] - ETA: 2s
4472832/11490434 [==========>...................] - ETA: 2s
5062656/11490434 [============>.................] - ETA: 1s
5652480/11490434 [=============>................] - ETA: 1s
6242304/11490434 [===============>..............] - ETA: 1s
6832128/11490434 [================>.............] - ETA: 1s
7421952/11490434 [==================>...........] - ETA: 0s
8011776/11490434 [===================>..........] - ETA: 0s
8568832/11490434 [=====================>........] - ETA: 0s
9076736/11490434 [======================>.......] - ETA: 0s
9633792/11490434 [========================>.....] - ETA: 0s
10223616/11490434 [=========================>....] - ETA: 0s
10813440/11490434 [===========================>..] - ETA: 0s
11403264/11490434 [============================>.] - ETA: 0s
11490434/11490434 [==============================] - 2s 0us/step
1.2. Model definition
Note that at this stage, there is nothing specific to the Akida IP. The model constructed below, as inspired by this example, is a completely standard Keras CNN model.
import keras
model_keras = keras.models.Sequential([
keras.layers.Rescaling(1. / 255, input_shape=(28, 28, 1)),
keras.layers.Conv2D(filters=32, kernel_size=3, strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
# Separable layer
keras.layers.DepthwiseConv2D(kernel_size=3, padding='same', strides=2),
keras.layers.Conv2D(filters=64, kernel_size=1, padding='same'),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
rescaling (Rescaling) (None, 28, 28, 1) 0
conv2d (Conv2D) (None, 13, 13, 32) 320
batch_normalization (BatchN (None, 13, 13, 32) 128
ormalization)
re_lu (ReLU) (None, 13, 13, 32) 0
depthwise_conv2d (Depthwise (None, 7, 7, 32) 320
Conv2D)
conv2d_1 (Conv2D) (None, 7, 7, 64) 2112
batch_normalization_1 (Batc (None, 7, 7, 64) 256
hNormalization)
re_lu_1 (ReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34,506
Trainable params: 34,314
Non-trainable params: 192
_________________________________________________________________
1.3. Model training
Given the model created above, train the model and check its accuracy. The model should achieve a test accuracy over 98% after 10 epochs.
from keras.optimizers import Adam
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=Adam(learning_rate=1e-3),
metrics=['accuracy'])
_ = model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
Epoch 1/10
1/1688 [..............................] - ETA: 19:04 - loss: 2.5054 - accuracy: 0.0938
30/1688 [..............................] - ETA: 2s - loss: 1.2525 - accuracy: 0.6094
60/1688 [>.............................] - ETA: 2s - loss: 0.8922 - accuracy: 0.7182
90/1688 [>.............................] - ETA: 2s - loss: 0.7141 - accuracy: 0.7802
121/1688 [=>............................] - ETA: 2s - loss: 0.6150 - accuracy: 0.8128
152/1688 [=>............................] - ETA: 2s - loss: 0.5530 - accuracy: 0.8335
184/1688 [==>...........................] - ETA: 2s - loss: 0.4988 - accuracy: 0.8507
215/1688 [==>...........................] - ETA: 2s - loss: 0.4602 - accuracy: 0.8624
245/1688 [===>..........................] - ETA: 2s - loss: 0.4331 - accuracy: 0.8712
276/1688 [===>..........................] - ETA: 2s - loss: 0.4074 - accuracy: 0.8793
307/1688 [====>.........................] - ETA: 2s - loss: 0.3860 - accuracy: 0.8857
336/1688 [====>.........................] - ETA: 2s - loss: 0.3685 - accuracy: 0.8909
366/1688 [=====>........................] - ETA: 2s - loss: 0.3564 - accuracy: 0.8946
396/1688 [======>.......................] - ETA: 2s - loss: 0.3414 - accuracy: 0.8988
425/1688 [======>.......................] - ETA: 2s - loss: 0.3294 - accuracy: 0.9026
454/1688 [=======>......................] - ETA: 2s - loss: 0.3186 - accuracy: 0.9058
484/1688 [=======>......................] - ETA: 2s - loss: 0.3091 - accuracy: 0.9087
514/1688 [========>.....................] - ETA: 1s - loss: 0.2989 - accuracy: 0.9118
545/1688 [========>.....................] - ETA: 1s - loss: 0.2887 - accuracy: 0.9147
575/1688 [=========>....................] - ETA: 1s - loss: 0.2814 - accuracy: 0.9175
605/1688 [=========>....................] - ETA: 1s - loss: 0.2734 - accuracy: 0.9197
636/1688 [==========>...................] - ETA: 1s - loss: 0.2676 - accuracy: 0.9216
667/1688 [==========>...................] - ETA: 1s - loss: 0.2633 - accuracy: 0.9227
697/1688 [===========>..................] - ETA: 1s - loss: 0.2572 - accuracy: 0.9244
728/1688 [===========>..................] - ETA: 1s - loss: 0.2516 - accuracy: 0.9257
759/1688 [============>.................] - ETA: 1s - loss: 0.2466 - accuracy: 0.9271
790/1688 [=============>................] - ETA: 1s - loss: 0.2415 - accuracy: 0.9287
821/1688 [=============>................] - ETA: 1s - loss: 0.2369 - accuracy: 0.9302
851/1688 [==============>...............] - ETA: 1s - loss: 0.2325 - accuracy: 0.9314
881/1688 [==============>...............] - ETA: 1s - loss: 0.2288 - accuracy: 0.9325
913/1688 [===============>..............] - ETA: 1s - loss: 0.2240 - accuracy: 0.9337
944/1688 [===============>..............] - ETA: 1s - loss: 0.2207 - accuracy: 0.9350
974/1688 [================>.............] - ETA: 1s - loss: 0.2180 - accuracy: 0.9357
1004/1688 [================>.............] - ETA: 1s - loss: 0.2158 - accuracy: 0.9363
1034/1688 [=================>............] - ETA: 1s - loss: 0.2122 - accuracy: 0.9374
1064/1688 [=================>............] - ETA: 1s - loss: 0.2096 - accuracy: 0.9381
1094/1688 [==================>...........] - ETA: 0s - loss: 0.2067 - accuracy: 0.9389
1124/1688 [==================>...........] - ETA: 0s - loss: 0.2042 - accuracy: 0.9396
1155/1688 [===================>..........] - ETA: 0s - loss: 0.2015 - accuracy: 0.9404
1186/1688 [====================>.........] - ETA: 0s - loss: 0.1992 - accuracy: 0.9412
1217/1688 [====================>.........] - ETA: 0s - loss: 0.1968 - accuracy: 0.9420
1248/1688 [=====================>........] - ETA: 0s - loss: 0.1947 - accuracy: 0.9427
1279/1688 [=====================>........] - ETA: 0s - loss: 0.1922 - accuracy: 0.9433
1310/1688 [======================>.......] - ETA: 0s - loss: 0.1906 - accuracy: 0.9438
1340/1688 [======================>.......] - ETA: 0s - loss: 0.1886 - accuracy: 0.9444
1370/1688 [=======================>......] - ETA: 0s - loss: 0.1868 - accuracy: 0.9449
1400/1688 [=======================>......] - ETA: 0s - loss: 0.1846 - accuracy: 0.9454
1430/1688 [========================>.....] - ETA: 0s - loss: 0.1834 - accuracy: 0.9458
1461/1688 [========================>.....] - ETA: 0s - loss: 0.1814 - accuracy: 0.9463
1491/1688 [=========================>....] - ETA: 0s - loss: 0.1801 - accuracy: 0.9467
1521/1688 [==========================>...] - ETA: 0s - loss: 0.1780 - accuracy: 0.9473
1551/1688 [==========================>...] - ETA: 0s - loss: 0.1766 - accuracy: 0.9478
1582/1688 [===========================>..] - ETA: 0s - loss: 0.1753 - accuracy: 0.9481
1612/1688 [===========================>..] - ETA: 0s - loss: 0.1739 - accuracy: 0.9485
1643/1688 [============================>.] - ETA: 0s - loss: 0.1720 - accuracy: 0.9491
1674/1688 [============================>.] - ETA: 0s - loss: 0.1701 - accuracy: 0.9497
1688/1688 [==============================] - ETA: 0s - loss: 0.1694 - accuracy: 0.9499
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1694 - accuracy: 0.9499 - val_loss: 0.1005 - val_accuracy: 0.9707
Epoch 2/10
1/1688 [..............................] - ETA: 2s - loss: 0.0946 - accuracy: 0.9688
33/1688 [..............................] - ETA: 2s - loss: 0.0810 - accuracy: 0.9744
64/1688 [>.............................] - ETA: 2s - loss: 0.0780 - accuracy: 0.9722
94/1688 [>.............................] - ETA: 2s - loss: 0.0698 - accuracy: 0.9774
124/1688 [=>............................] - ETA: 2s - loss: 0.0698 - accuracy: 0.9786
154/1688 [=>............................] - ETA: 2s - loss: 0.0701 - accuracy: 0.9781
184/1688 [==>...........................] - ETA: 2s - loss: 0.0704 - accuracy: 0.9784
214/1688 [==>...........................] - ETA: 2s - loss: 0.0712 - accuracy: 0.9782
245/1688 [===>..........................] - ETA: 2s - loss: 0.0701 - accuracy: 0.9784
276/1688 [===>..........................] - ETA: 2s - loss: 0.0677 - accuracy: 0.9787
306/1688 [====>.........................] - ETA: 2s - loss: 0.0691 - accuracy: 0.9785
336/1688 [====>.........................] - ETA: 2s - loss: 0.0691 - accuracy: 0.9782
366/1688 [=====>........................] - ETA: 2s - loss: 0.0688 - accuracy: 0.9783
397/1688 [======>.......................] - ETA: 2s - loss: 0.0693 - accuracy: 0.9780
427/1688 [======>.......................] - ETA: 2s - loss: 0.0688 - accuracy: 0.9783
457/1688 [=======>......................] - ETA: 2s - loss: 0.0689 - accuracy: 0.9783
488/1688 [=======>......................] - ETA: 2s - loss: 0.0695 - accuracy: 0.9780
518/1688 [========>.....................] - ETA: 1s - loss: 0.0686 - accuracy: 0.9785
548/1688 [========>.....................] - ETA: 1s - loss: 0.0710 - accuracy: 0.9780
579/1688 [=========>....................] - ETA: 1s - loss: 0.0716 - accuracy: 0.9781
609/1688 [=========>....................] - ETA: 1s - loss: 0.0714 - accuracy: 0.9782
639/1688 [==========>...................] - ETA: 1s - loss: 0.0728 - accuracy: 0.9779
669/1688 [==========>...................] - ETA: 1s - loss: 0.0738 - accuracy: 0.9775
699/1688 [===========>..................] - ETA: 1s - loss: 0.0751 - accuracy: 0.9772
729/1688 [===========>..................] - ETA: 1s - loss: 0.0750 - accuracy: 0.9771
759/1688 [============>.................] - ETA: 1s - loss: 0.0743 - accuracy: 0.9773
790/1688 [=============>................] - ETA: 1s - loss: 0.0742 - accuracy: 0.9774
820/1688 [=============>................] - ETA: 1s - loss: 0.0751 - accuracy: 0.9771
850/1688 [==============>...............] - ETA: 1s - loss: 0.0746 - accuracy: 0.9773
880/1688 [==============>...............] - ETA: 1s - loss: 0.0750 - accuracy: 0.9772
910/1688 [===============>..............] - ETA: 1s - loss: 0.0745 - accuracy: 0.9772
941/1688 [===============>..............] - ETA: 1s - loss: 0.0744 - accuracy: 0.9774
971/1688 [================>.............] - ETA: 1s - loss: 0.0740 - accuracy: 0.9775
1001/1688 [================>.............] - ETA: 1s - loss: 0.0736 - accuracy: 0.9776
1031/1688 [=================>............] - ETA: 1s - loss: 0.0732 - accuracy: 0.9776
1061/1688 [=================>............] - ETA: 1s - loss: 0.0729 - accuracy: 0.9776
1091/1688 [==================>...........] - ETA: 1s - loss: 0.0728 - accuracy: 0.9778
1121/1688 [==================>...........] - ETA: 0s - loss: 0.0727 - accuracy: 0.9778
1151/1688 [===================>..........] - ETA: 0s - loss: 0.0722 - accuracy: 0.9780
1182/1688 [====================>.........] - ETA: 0s - loss: 0.0723 - accuracy: 0.9780
1213/1688 [====================>.........] - ETA: 0s - loss: 0.0723 - accuracy: 0.9779
1243/1688 [=====================>........] - ETA: 0s - loss: 0.0719 - accuracy: 0.9780
1273/1688 [=====================>........] - ETA: 0s - loss: 0.0731 - accuracy: 0.9775
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0735 - accuracy: 0.9774
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0739 - accuracy: 0.9773
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0739 - accuracy: 0.9771
1396/1688 [=======================>......] - ETA: 0s - loss: 0.0738 - accuracy: 0.9771
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0731 - accuracy: 0.9774
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0727 - accuracy: 0.9775
1486/1688 [=========================>....] - ETA: 0s - loss: 0.0724 - accuracy: 0.9776
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0722 - accuracy: 0.9777
1546/1688 [==========================>...] - ETA: 0s - loss: 0.0725 - accuracy: 0.9775
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0724 - accuracy: 0.9776
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0720 - accuracy: 0.9777
1638/1688 [============================>.] - ETA: 0s - loss: 0.0721 - accuracy: 0.9777
1668/1688 [============================>.] - ETA: 0s - loss: 0.0723 - accuracy: 0.9776
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0721 - accuracy: 0.9777 - val_loss: 0.0540 - val_accuracy: 0.9855
Epoch 3/10
1/1688 [..............................] - ETA: 2s - loss: 0.0072 - accuracy: 1.0000
35/1688 [..............................] - ETA: 2s - loss: 0.0476 - accuracy: 0.9884
68/1688 [>.............................] - ETA: 2s - loss: 0.0457 - accuracy: 0.9876
102/1688 [>.............................] - ETA: 2s - loss: 0.0410 - accuracy: 0.9887
133/1688 [=>............................] - ETA: 2s - loss: 0.0399 - accuracy: 0.9885
164/1688 [=>............................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9865
194/1688 [==>...........................] - ETA: 2s - loss: 0.0454 - accuracy: 0.9861
224/1688 [==>...........................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9860
255/1688 [===>..........................] - ETA: 2s - loss: 0.0442 - accuracy: 0.9863
285/1688 [====>.........................] - ETA: 2s - loss: 0.0464 - accuracy: 0.9859
315/1688 [====>.........................] - ETA: 2s - loss: 0.0459 - accuracy: 0.9863
345/1688 [=====>........................] - ETA: 2s - loss: 0.0470 - accuracy: 0.9860
375/1688 [=====>........................] - ETA: 2s - loss: 0.0477 - accuracy: 0.9853
406/1688 [======>.......................] - ETA: 2s - loss: 0.0495 - accuracy: 0.9850
436/1688 [======>.......................] - ETA: 2s - loss: 0.0486 - accuracy: 0.9852
465/1688 [=======>......................] - ETA: 2s - loss: 0.0483 - accuracy: 0.9854
495/1688 [=======>......................] - ETA: 1s - loss: 0.0493 - accuracy: 0.9850
526/1688 [========>.....................] - ETA: 1s - loss: 0.0496 - accuracy: 0.9851
556/1688 [========>.....................] - ETA: 1s - loss: 0.0502 - accuracy: 0.9848
586/1688 [=========>....................] - ETA: 1s - loss: 0.0510 - accuracy: 0.9844
617/1688 [=========>....................] - ETA: 1s - loss: 0.0506 - accuracy: 0.9846
648/1688 [==========>...................] - ETA: 1s - loss: 0.0507 - accuracy: 0.9843
679/1688 [===========>..................] - ETA: 1s - loss: 0.0504 - accuracy: 0.9842
710/1688 [===========>..................] - ETA: 1s - loss: 0.0508 - accuracy: 0.9839
741/1688 [============>.................] - ETA: 1s - loss: 0.0513 - accuracy: 0.9838
771/1688 [============>.................] - ETA: 1s - loss: 0.0514 - accuracy: 0.9837
801/1688 [=============>................] - ETA: 1s - loss: 0.0513 - accuracy: 0.9838
831/1688 [=============>................] - ETA: 1s - loss: 0.0515 - accuracy: 0.9837
862/1688 [==============>...............] - ETA: 1s - loss: 0.0516 - accuracy: 0.9838
892/1688 [==============>...............] - ETA: 1s - loss: 0.0508 - accuracy: 0.9840
923/1688 [===============>..............] - ETA: 1s - loss: 0.0507 - accuracy: 0.9841
954/1688 [===============>..............] - ETA: 1s - loss: 0.0502 - accuracy: 0.9842
984/1688 [================>.............] - ETA: 1s - loss: 0.0500 - accuracy: 0.9842
1014/1688 [=================>............] - ETA: 1s - loss: 0.0500 - accuracy: 0.9842
1045/1688 [=================>............] - ETA: 1s - loss: 0.0498 - accuracy: 0.9842
1076/1688 [==================>...........] - ETA: 1s - loss: 0.0495 - accuracy: 0.9843
1106/1688 [==================>...........] - ETA: 0s - loss: 0.0492 - accuracy: 0.9844
1137/1688 [===================>..........] - ETA: 0s - loss: 0.0495 - accuracy: 0.9844
1167/1688 [===================>..........] - ETA: 0s - loss: 0.0503 - accuracy: 0.9841
1198/1688 [====================>.........] - ETA: 0s - loss: 0.0505 - accuracy: 0.9841
1228/1688 [====================>.........] - ETA: 0s - loss: 0.0502 - accuracy: 0.9842
1259/1688 [=====================>........] - ETA: 0s - loss: 0.0504 - accuracy: 0.9842
1290/1688 [=====================>........] - ETA: 0s - loss: 0.0507 - accuracy: 0.9842
1321/1688 [======================>.......] - ETA: 0s - loss: 0.0509 - accuracy: 0.9843
1352/1688 [=======================>......] - ETA: 0s - loss: 0.0514 - accuracy: 0.9842
1382/1688 [=======================>......] - ETA: 0s - loss: 0.0509 - accuracy: 0.9844
1412/1688 [========================>.....] - ETA: 0s - loss: 0.0508 - accuracy: 0.9843
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0504 - accuracy: 0.9844
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0500 - accuracy: 0.9844
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0503 - accuracy: 0.9844
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0503 - accuracy: 0.9843
1564/1688 [==========================>...] - ETA: 0s - loss: 0.0506 - accuracy: 0.9841
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0503 - accuracy: 0.9842
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0507 - accuracy: 0.9841
1654/1688 [============================>.] - ETA: 0s - loss: 0.0513 - accuracy: 0.9840
1684/1688 [============================>.] - ETA: 0s - loss: 0.0516 - accuracy: 0.9839
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0515 - accuracy: 0.9839 - val_loss: 0.0502 - val_accuracy: 0.9873
Epoch 4/10
1/1688 [..............................] - ETA: 2s - loss: 0.1013 - accuracy: 0.9688
32/1688 [..............................] - ETA: 2s - loss: 0.0474 - accuracy: 0.9883
62/1688 [>.............................] - ETA: 2s - loss: 0.0400 - accuracy: 0.9904
93/1688 [>.............................] - ETA: 2s - loss: 0.0423 - accuracy: 0.9886
123/1688 [=>............................] - ETA: 2s - loss: 0.0462 - accuracy: 0.9873
153/1688 [=>............................] - ETA: 2s - loss: 0.0475 - accuracy: 0.9867
184/1688 [==>...........................] - ETA: 2s - loss: 0.0446 - accuracy: 0.9869
215/1688 [==>...........................] - ETA: 2s - loss: 0.0431 - accuracy: 0.9876
246/1688 [===>..........................] - ETA: 2s - loss: 0.0435 - accuracy: 0.9873
277/1688 [===>..........................] - ETA: 2s - loss: 0.0456 - accuracy: 0.9863
307/1688 [====>.........................] - ETA: 2s - loss: 0.0465 - accuracy: 0.9864
338/1688 [=====>........................] - ETA: 2s - loss: 0.0453 - accuracy: 0.9865
369/1688 [=====>........................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9864
400/1688 [======>.......................] - ETA: 2s - loss: 0.0441 - accuracy: 0.9867
431/1688 [======>.......................] - ETA: 2s - loss: 0.0441 - accuracy: 0.9869
462/1688 [=======>......................] - ETA: 2s - loss: 0.0428 - accuracy: 0.9873
492/1688 [=======>......................] - ETA: 1s - loss: 0.0424 - accuracy: 0.9874
523/1688 [========>.....................] - ETA: 1s - loss: 0.0419 - accuracy: 0.9873
553/1688 [========>.....................] - ETA: 1s - loss: 0.0414 - accuracy: 0.9873
583/1688 [=========>....................] - ETA: 1s - loss: 0.0411 - accuracy: 0.9873
614/1688 [=========>....................] - ETA: 1s - loss: 0.0409 - accuracy: 0.9874
644/1688 [==========>...................] - ETA: 1s - loss: 0.0402 - accuracy: 0.9875
675/1688 [==========>...................] - ETA: 1s - loss: 0.0395 - accuracy: 0.9877
706/1688 [===========>..................] - ETA: 1s - loss: 0.0394 - accuracy: 0.9877
736/1688 [============>.................] - ETA: 1s - loss: 0.0398 - accuracy: 0.9876
766/1688 [============>.................] - ETA: 1s - loss: 0.0408 - accuracy: 0.9875
796/1688 [=============>................] - ETA: 1s - loss: 0.0409 - accuracy: 0.9875
826/1688 [=============>................] - ETA: 1s - loss: 0.0407 - accuracy: 0.9876
856/1688 [==============>...............] - ETA: 1s - loss: 0.0415 - accuracy: 0.9874
887/1688 [==============>...............] - ETA: 1s - loss: 0.0416 - accuracy: 0.9873
917/1688 [===============>..............] - ETA: 1s - loss: 0.0421 - accuracy: 0.9871
947/1688 [===============>..............] - ETA: 1s - loss: 0.0416 - accuracy: 0.9872
977/1688 [================>.............] - ETA: 1s - loss: 0.0415 - accuracy: 0.9873
1007/1688 [================>.............] - ETA: 1s - loss: 0.0411 - accuracy: 0.9874
1038/1688 [=================>............] - ETA: 1s - loss: 0.0418 - accuracy: 0.9871
1069/1688 [=================>............] - ETA: 1s - loss: 0.0416 - accuracy: 0.9871
1100/1688 [==================>...........] - ETA: 0s - loss: 0.0423 - accuracy: 0.9868
1130/1688 [===================>..........] - ETA: 0s - loss: 0.0425 - accuracy: 0.9868
1161/1688 [===================>..........] - ETA: 0s - loss: 0.0434 - accuracy: 0.9865
1192/1688 [====================>.........] - ETA: 0s - loss: 0.0433 - accuracy: 0.9866
1223/1688 [====================>.........] - ETA: 0s - loss: 0.0436 - accuracy: 0.9865
1254/1688 [=====================>........] - ETA: 0s - loss: 0.0434 - accuracy: 0.9866
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0434 - accuracy: 0.9866
1316/1688 [======================>.......] - ETA: 0s - loss: 0.0432 - accuracy: 0.9866
1347/1688 [======================>.......] - ETA: 0s - loss: 0.0428 - accuracy: 0.9865
1377/1688 [=======================>......] - ETA: 0s - loss: 0.0432 - accuracy: 0.9864
1408/1688 [========================>.....] - ETA: 0s - loss: 0.0430 - accuracy: 0.9865
1439/1688 [========================>.....] - ETA: 0s - loss: 0.0428 - accuracy: 0.9866
1469/1688 [=========================>....] - ETA: 0s - loss: 0.0432 - accuracy: 0.9864
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0431 - accuracy: 0.9865
1529/1688 [==========================>...] - ETA: 0s - loss: 0.0431 - accuracy: 0.9865
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0431 - accuracy: 0.9864
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0433 - accuracy: 0.9863
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0433 - accuracy: 0.9863
1650/1688 [============================>.] - ETA: 0s - loss: 0.0435 - accuracy: 0.9863
1681/1688 [============================>.] - ETA: 0s - loss: 0.0437 - accuracy: 0.9863
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0435 - accuracy: 0.9863 - val_loss: 0.0924 - val_accuracy: 0.9777
Epoch 5/10
1/1688 [..............................] - ETA: 2s - loss: 0.0226 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9934
64/1688 [>.............................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9932
95/1688 [>.............................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9928
126/1688 [=>............................] - ETA: 2s - loss: 0.0224 - accuracy: 0.9916
156/1688 [=>............................] - ETA: 2s - loss: 0.0254 - accuracy: 0.9910
186/1688 [==>...........................] - ETA: 2s - loss: 0.0299 - accuracy: 0.9899
216/1688 [==>...........................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9909
246/1688 [===>..........................] - ETA: 2s - loss: 0.0278 - accuracy: 0.9903
276/1688 [===>..........................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9908
306/1688 [====>.........................] - ETA: 2s - loss: 0.0256 - accuracy: 0.9911
336/1688 [====>.........................] - ETA: 2s - loss: 0.0251 - accuracy: 0.9914
366/1688 [=====>........................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9915
396/1688 [======>.......................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9915
426/1688 [======>.......................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9916
456/1688 [=======>......................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9914
486/1688 [=======>......................] - ETA: 2s - loss: 0.0266 - accuracy: 0.9912
516/1688 [========>.....................] - ETA: 1s - loss: 0.0269 - accuracy: 0.9910
546/1688 [========>.....................] - ETA: 1s - loss: 0.0274 - accuracy: 0.9910
577/1688 [=========>....................] - ETA: 1s - loss: 0.0282 - accuracy: 0.9907
607/1688 [=========>....................] - ETA: 1s - loss: 0.0293 - accuracy: 0.9903
637/1688 [==========>...................] - ETA: 1s - loss: 0.0286 - accuracy: 0.9906
668/1688 [==========>...................] - ETA: 1s - loss: 0.0291 - accuracy: 0.9905
698/1688 [===========>..................] - ETA: 1s - loss: 0.0293 - accuracy: 0.9904
729/1688 [===========>..................] - ETA: 1s - loss: 0.0298 - accuracy: 0.9903
759/1688 [============>.................] - ETA: 1s - loss: 0.0301 - accuracy: 0.9903
789/1688 [=============>................] - ETA: 1s - loss: 0.0306 - accuracy: 0.9901
820/1688 [=============>................] - ETA: 1s - loss: 0.0316 - accuracy: 0.9898
851/1688 [==============>...............] - ETA: 1s - loss: 0.0320 - accuracy: 0.9897
881/1688 [==============>...............] - ETA: 1s - loss: 0.0324 - accuracy: 0.9895
912/1688 [===============>..............] - ETA: 1s - loss: 0.0334 - accuracy: 0.9893
942/1688 [===============>..............] - ETA: 1s - loss: 0.0344 - accuracy: 0.9890
972/1688 [================>.............] - ETA: 1s - loss: 0.0345 - accuracy: 0.9890
1002/1688 [================>.............] - ETA: 1s - loss: 0.0343 - accuracy: 0.9891
1032/1688 [=================>............] - ETA: 1s - loss: 0.0340 - accuracy: 0.9891
1062/1688 [=================>............] - ETA: 1s - loss: 0.0344 - accuracy: 0.9889
1092/1688 [==================>...........] - ETA: 0s - loss: 0.0347 - accuracy: 0.9888
1122/1688 [==================>...........] - ETA: 0s - loss: 0.0346 - accuracy: 0.9888
1152/1688 [===================>..........] - ETA: 0s - loss: 0.0351 - accuracy: 0.9888
1183/1688 [====================>.........] - ETA: 0s - loss: 0.0348 - accuracy: 0.9889
1212/1688 [====================>.........] - ETA: 0s - loss: 0.0349 - accuracy: 0.9889
1243/1688 [=====================>........] - ETA: 0s - loss: 0.0347 - accuracy: 0.9889
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0350 - accuracy: 0.9888
1305/1688 [======================>.......] - ETA: 0s - loss: 0.0352 - accuracy: 0.9887
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0353 - accuracy: 0.9887
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0354 - accuracy: 0.9886
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0354 - accuracy: 0.9886
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0351 - accuracy: 0.9888
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0351 - accuracy: 0.9888
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0350 - accuracy: 0.9888
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0350 - accuracy: 0.9889
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0349 - accuracy: 0.9889
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0350 - accuracy: 0.9889
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0353 - accuracy: 0.9888
1637/1688 [============================>.] - ETA: 0s - loss: 0.0351 - accuracy: 0.9888
1668/1688 [============================>.] - ETA: 0s - loss: 0.0353 - accuracy: 0.9888
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0355 - accuracy: 0.9887 - val_loss: 0.0571 - val_accuracy: 0.9850
Epoch 6/10
1/1688 [..............................] - ETA: 2s - loss: 0.0559 - accuracy: 0.9688
34/1688 [..............................] - ETA: 2s - loss: 0.0121 - accuracy: 0.9982
65/1688 [>.............................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9952
96/1688 [>.............................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9945
126/1688 [=>............................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9940
157/1688 [=>............................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9940
188/1688 [==>...........................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9937
218/1688 [==>...........................] - ETA: 2s - loss: 0.0227 - accuracy: 0.9930
249/1688 [===>..........................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9933
279/1688 [===>..........................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9936
309/1688 [====>.........................] - ETA: 2s - loss: 0.0197 - accuracy: 0.9938
340/1688 [=====>........................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9934
371/1688 [=====>........................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9928
401/1688 [======>.......................] - ETA: 2s - loss: 0.0223 - accuracy: 0.9929
432/1688 [======>.......................] - ETA: 2s - loss: 0.0221 - accuracy: 0.9927
462/1688 [=======>......................] - ETA: 2s - loss: 0.0236 - accuracy: 0.9925
492/1688 [=======>......................] - ETA: 1s - loss: 0.0241 - accuracy: 0.9923
522/1688 [========>.....................] - ETA: 1s - loss: 0.0240 - accuracy: 0.9923
553/1688 [========>.....................] - ETA: 1s - loss: 0.0242 - accuracy: 0.9923
583/1688 [=========>....................] - ETA: 1s - loss: 0.0244 - accuracy: 0.9923
614/1688 [=========>....................] - ETA: 1s - loss: 0.0240 - accuracy: 0.9925
645/1688 [==========>...................] - ETA: 1s - loss: 0.0236 - accuracy: 0.9926
676/1688 [===========>..................] - ETA: 1s - loss: 0.0233 - accuracy: 0.9926
706/1688 [===========>..................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9923
737/1688 [============>.................] - ETA: 1s - loss: 0.0234 - accuracy: 0.9924
767/1688 [============>.................] - ETA: 1s - loss: 0.0236 - accuracy: 0.9924
798/1688 [=============>................] - ETA: 1s - loss: 0.0242 - accuracy: 0.9922
828/1688 [=============>................] - ETA: 1s - loss: 0.0242 - accuracy: 0.9921
858/1688 [==============>...............] - ETA: 1s - loss: 0.0247 - accuracy: 0.9920
888/1688 [==============>...............] - ETA: 1s - loss: 0.0251 - accuracy: 0.9919
918/1688 [===============>..............] - ETA: 1s - loss: 0.0251 - accuracy: 0.9918
948/1688 [===============>..............] - ETA: 1s - loss: 0.0251 - accuracy: 0.9918
978/1688 [================>.............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9917
1008/1688 [================>.............] - ETA: 1s - loss: 0.0258 - accuracy: 0.9915
1038/1688 [=================>............] - ETA: 1s - loss: 0.0261 - accuracy: 0.9912
1069/1688 [=================>............] - ETA: 1s - loss: 0.0266 - accuracy: 0.9911
1100/1688 [==================>...........] - ETA: 0s - loss: 0.0265 - accuracy: 0.9910
1131/1688 [===================>..........] - ETA: 0s - loss: 0.0268 - accuracy: 0.9910
1162/1688 [===================>..........] - ETA: 0s - loss: 0.0273 - accuracy: 0.9909
1193/1688 [====================>.........] - ETA: 0s - loss: 0.0279 - accuracy: 0.9908
1223/1688 [====================>.........] - ETA: 0s - loss: 0.0280 - accuracy: 0.9908
1253/1688 [=====================>........] - ETA: 0s - loss: 0.0283 - accuracy: 0.9907
1284/1688 [=====================>........] - ETA: 0s - loss: 0.0282 - accuracy: 0.9907
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0286 - accuracy: 0.9906
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0290 - accuracy: 0.9905
1374/1688 [=======================>......] - ETA: 0s - loss: 0.0298 - accuracy: 0.9903
1405/1688 [=======================>......] - ETA: 0s - loss: 0.0301 - accuracy: 0.9902
1436/1688 [========================>.....] - ETA: 0s - loss: 0.0299 - accuracy: 0.9903
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0301 - accuracy: 0.9902
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0300 - accuracy: 0.9903
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0301 - accuracy: 0.9902
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0300 - accuracy: 0.9903
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0302 - accuracy: 0.9902
1617/1688 [===========================>..] - ETA: 0s - loss: 0.0305 - accuracy: 0.9902
1648/1688 [============================>.] - ETA: 0s - loss: 0.0303 - accuracy: 0.9902
1679/1688 [============================>.] - ETA: 0s - loss: 0.0301 - accuracy: 0.9903
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0301 - accuracy: 0.9903 - val_loss: 0.0563 - val_accuracy: 0.9857
Epoch 7/10
1/1688 [..............................] - ETA: 2s - loss: 2.7734e-04 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0287 - accuracy: 0.9883
62/1688 [>.............................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9909
93/1688 [>.............................] - ETA: 2s - loss: 0.0283 - accuracy: 0.9909
124/1688 [=>............................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9912
154/1688 [=>............................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9919
184/1688 [==>...........................] - ETA: 2s - loss: 0.0252 - accuracy: 0.9910
214/1688 [==>...........................] - ETA: 2s - loss: 0.0253 - accuracy: 0.9905
244/1688 [===>..........................] - ETA: 2s - loss: 0.0251 - accuracy: 0.9903
274/1688 [===>..........................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9908
304/1688 [====>.........................] - ETA: 2s - loss: 0.0241 - accuracy: 0.9907
334/1688 [====>.........................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9912
365/1688 [=====>........................] - ETA: 2s - loss: 0.0223 - accuracy: 0.9915
395/1688 [======>.......................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9918
425/1688 [======>.......................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9917
456/1688 [=======>......................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9917
486/1688 [=======>......................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9920
517/1688 [========>.....................] - ETA: 1s - loss: 0.0212 - accuracy: 0.9920
547/1688 [========>.....................] - ETA: 1s - loss: 0.0213 - accuracy: 0.9920
578/1688 [=========>....................] - ETA: 1s - loss: 0.0222 - accuracy: 0.9918
608/1688 [=========>....................] - ETA: 1s - loss: 0.0228 - accuracy: 0.9918
638/1688 [==========>...................] - ETA: 1s - loss: 0.0225 - accuracy: 0.9918
668/1688 [==========>...................] - ETA: 1s - loss: 0.0222 - accuracy: 0.9919
698/1688 [===========>..................] - ETA: 1s - loss: 0.0221 - accuracy: 0.9919
728/1688 [===========>..................] - ETA: 1s - loss: 0.0227 - accuracy: 0.9917
759/1688 [============>.................] - ETA: 1s - loss: 0.0227 - accuracy: 0.9917
789/1688 [=============>................] - ETA: 1s - loss: 0.0227 - accuracy: 0.9917
819/1688 [=============>................] - ETA: 1s - loss: 0.0226 - accuracy: 0.9918
849/1688 [==============>...............] - ETA: 1s - loss: 0.0224 - accuracy: 0.9919
880/1688 [==============>...............] - ETA: 1s - loss: 0.0225 - accuracy: 0.9920
911/1688 [===============>..............] - ETA: 1s - loss: 0.0230 - accuracy: 0.9919
941/1688 [===============>..............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9918
971/1688 [================>.............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9917
1001/1688 [================>.............] - ETA: 1s - loss: 0.0238 - accuracy: 0.9915
1032/1688 [=================>............] - ETA: 1s - loss: 0.0237 - accuracy: 0.9915
1062/1688 [=================>............] - ETA: 1s - loss: 0.0238 - accuracy: 0.9914
1092/1688 [==================>...........] - ETA: 1s - loss: 0.0240 - accuracy: 0.9912
1122/1688 [==================>...........] - ETA: 0s - loss: 0.0238 - accuracy: 0.9913
1152/1688 [===================>..........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9912
1182/1688 [====================>.........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9912
1212/1688 [====================>.........] - ETA: 0s - loss: 0.0238 - accuracy: 0.9913
1243/1688 [=====================>........] - ETA: 0s - loss: 0.0242 - accuracy: 0.9912
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0244 - accuracy: 0.9911
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0246 - accuracy: 0.9912
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0245 - accuracy: 0.9912
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0245 - accuracy: 0.9912
1396/1688 [=======================>......] - ETA: 0s - loss: 0.0248 - accuracy: 0.9912
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0246 - accuracy: 0.9913
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0247 - accuracy: 0.9912
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0248 - accuracy: 0.9912
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0251 - accuracy: 0.9911
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0248 - accuracy: 0.9913
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0252 - accuracy: 0.9912
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0254 - accuracy: 0.9911
1637/1688 [============================>.] - ETA: 0s - loss: 0.0253 - accuracy: 0.9911
1668/1688 [============================>.] - ETA: 0s - loss: 0.0256 - accuracy: 0.9909
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0257 - accuracy: 0.9909 - val_loss: 0.0532 - val_accuracy: 0.9873
Epoch 8/10
1/1688 [..............................] - ETA: 2s - loss: 0.0413 - accuracy: 0.9688
34/1688 [..............................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9917
64/1688 [>.............................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9937
95/1688 [>.............................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9934
126/1688 [=>............................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9936
156/1688 [=>............................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9936
187/1688 [==>...........................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9938
218/1688 [==>...........................] - ETA: 2s - loss: 0.0187 - accuracy: 0.9938
250/1688 [===>..........................] - ETA: 2s - loss: 0.0175 - accuracy: 0.9942
282/1688 [====>.........................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9938
313/1688 [====>.........................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9938
343/1688 [=====>........................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9941
373/1688 [=====>........................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9938
403/1688 [======>.......................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9939
433/1688 [======>.......................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9940
463/1688 [=======>......................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9941
494/1688 [=======>......................] - ETA: 1s - loss: 0.0190 - accuracy: 0.9940
524/1688 [========>.....................] - ETA: 1s - loss: 0.0185 - accuracy: 0.9942
554/1688 [========>.....................] - ETA: 1s - loss: 0.0188 - accuracy: 0.9941
584/1688 [=========>....................] - ETA: 1s - loss: 0.0190 - accuracy: 0.9941
614/1688 [=========>....................] - ETA: 1s - loss: 0.0195 - accuracy: 0.9938
644/1688 [==========>...................] - ETA: 1s - loss: 0.0198 - accuracy: 0.9937
674/1688 [==========>...................] - ETA: 1s - loss: 0.0197 - accuracy: 0.9937
704/1688 [===========>..................] - ETA: 1s - loss: 0.0195 - accuracy: 0.9937
735/1688 [============>.................] - ETA: 1s - loss: 0.0193 - accuracy: 0.9937
765/1688 [============>.................] - ETA: 1s - loss: 0.0195 - accuracy: 0.9936
796/1688 [=============>................] - ETA: 1s - loss: 0.0193 - accuracy: 0.9936
826/1688 [=============>................] - ETA: 1s - loss: 0.0192 - accuracy: 0.9937
855/1688 [==============>...............] - ETA: 1s - loss: 0.0191 - accuracy: 0.9938
885/1688 [==============>...............] - ETA: 1s - loss: 0.0193 - accuracy: 0.9936
915/1688 [===============>..............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9934
946/1688 [===============>..............] - ETA: 1s - loss: 0.0197 - accuracy: 0.9933
976/1688 [================>.............] - ETA: 1s - loss: 0.0200 - accuracy: 0.9933
1006/1688 [================>.............] - ETA: 1s - loss: 0.0201 - accuracy: 0.9933
1037/1688 [=================>............] - ETA: 1s - loss: 0.0200 - accuracy: 0.9933
1067/1688 [=================>............] - ETA: 1s - loss: 0.0198 - accuracy: 0.9934
1097/1688 [==================>...........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9934
1127/1688 [===================>..........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9933
1156/1688 [===================>..........] - ETA: 0s - loss: 0.0200 - accuracy: 0.9932
1186/1688 [====================>.........] - ETA: 0s - loss: 0.0201 - accuracy: 0.9931
1216/1688 [====================>.........] - ETA: 0s - loss: 0.0201 - accuracy: 0.9932
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0201 - accuracy: 0.9932
1276/1688 [=====================>........] - ETA: 0s - loss: 0.0204 - accuracy: 0.9931
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0207 - accuracy: 0.9931
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0209 - accuracy: 0.9929
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0210 - accuracy: 0.9929
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0210 - accuracy: 0.9929
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0208 - accuracy: 0.9930
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0206 - accuracy: 0.9930
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0208 - accuracy: 0.9930
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0209 - accuracy: 0.9929
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0209 - accuracy: 0.9929
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0209 - accuracy: 0.9929
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0211 - accuracy: 0.9928
1643/1688 [============================>.] - ETA: 0s - loss: 0.0214 - accuracy: 0.9927
1674/1688 [============================>.] - ETA: 0s - loss: 0.0215 - accuracy: 0.9927
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0214 - accuracy: 0.9927 - val_loss: 0.0685 - val_accuracy: 0.9828
Epoch 9/10
1/1688 [..............................] - ETA: 2s - loss: 0.0027 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9953
63/1688 [>.............................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9921
94/1688 [>.............................] - ETA: 2s - loss: 0.0229 - accuracy: 0.9914
124/1688 [=>............................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9917
155/1688 [=>............................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9929
185/1688 [==>...........................] - ETA: 2s - loss: 0.0186 - accuracy: 0.9936
214/1688 [==>...........................] - ETA: 2s - loss: 0.0187 - accuracy: 0.9936
244/1688 [===>..........................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9937
274/1688 [===>..........................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9937
305/1688 [====>.........................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9941
335/1688 [====>.........................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9941
365/1688 [=====>........................] - ETA: 2s - loss: 0.0158 - accuracy: 0.9943
395/1688 [======>.......................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9944
424/1688 [======>.......................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9946
453/1688 [=======>......................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9946
481/1688 [=======>......................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9945
510/1688 [========>.....................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9945
539/1688 [========>.....................] - ETA: 1s - loss: 0.0163 - accuracy: 0.9941
567/1688 [=========>....................] - ETA: 1s - loss: 0.0162 - accuracy: 0.9942
596/1688 [=========>....................] - ETA: 1s - loss: 0.0168 - accuracy: 0.9940
625/1688 [==========>...................] - ETA: 1s - loss: 0.0169 - accuracy: 0.9938
653/1688 [==========>...................] - ETA: 1s - loss: 0.0170 - accuracy: 0.9939
682/1688 [===========>..................] - ETA: 1s - loss: 0.0175 - accuracy: 0.9936
711/1688 [===========>..................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9935
740/1688 [============>.................] - ETA: 1s - loss: 0.0183 - accuracy: 0.9934
768/1688 [============>.................] - ETA: 1s - loss: 0.0185 - accuracy: 0.9934
797/1688 [=============>................] - ETA: 1s - loss: 0.0183 - accuracy: 0.9935
826/1688 [=============>................] - ETA: 1s - loss: 0.0182 - accuracy: 0.9934
855/1688 [==============>...............] - ETA: 1s - loss: 0.0184 - accuracy: 0.9933
883/1688 [==============>...............] - ETA: 1s - loss: 0.0186 - accuracy: 0.9932
911/1688 [===============>..............] - ETA: 1s - loss: 0.0184 - accuracy: 0.9933
940/1688 [===============>..............] - ETA: 1s - loss: 0.0184 - accuracy: 0.9933
969/1688 [================>.............] - ETA: 1s - loss: 0.0184 - accuracy: 0.9933
999/1688 [================>.............] - ETA: 1s - loss: 0.0182 - accuracy: 0.9934
1030/1688 [=================>............] - ETA: 1s - loss: 0.0181 - accuracy: 0.9935
1061/1688 [=================>............] - ETA: 1s - loss: 0.0183 - accuracy: 0.9934
1091/1688 [==================>...........] - ETA: 1s - loss: 0.0183 - accuracy: 0.9933
1122/1688 [==================>...........] - ETA: 0s - loss: 0.0181 - accuracy: 0.9934
1152/1688 [===================>..........] - ETA: 0s - loss: 0.0180 - accuracy: 0.9935
1182/1688 [====================>.........] - ETA: 0s - loss: 0.0179 - accuracy: 0.9935
1212/1688 [====================>.........] - ETA: 0s - loss: 0.0177 - accuracy: 0.9936
1243/1688 [=====================>........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9935
1273/1688 [=====================>........] - ETA: 0s - loss: 0.0177 - accuracy: 0.9935
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0177 - accuracy: 0.9935
1333/1688 [======================>.......] - ETA: 0s - loss: 0.0178 - accuracy: 0.9935
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0183 - accuracy: 0.9934
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0181 - accuracy: 0.9934
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0183 - accuracy: 0.9933
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0184 - accuracy: 0.9933
1486/1688 [=========================>....] - ETA: 0s - loss: 0.0187 - accuracy: 0.9931
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0189 - accuracy: 0.9931
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0190 - accuracy: 0.9930
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0189 - accuracy: 0.9931
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0188 - accuracy: 0.9931
1639/1688 [============================>.] - ETA: 0s - loss: 0.0189 - accuracy: 0.9931
1670/1688 [============================>.] - ETA: 0s - loss: 0.0190 - accuracy: 0.9931
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0190 - accuracy: 0.9931 - val_loss: 0.0567 - val_accuracy: 0.9862
Epoch 10/10
1/1688 [..............................] - ETA: 2s - loss: 0.0620 - accuracy: 0.9375
33/1688 [..............................] - ETA: 2s - loss: 0.0169 - accuracy: 0.9915
64/1688 [>.............................] - ETA: 2s - loss: 0.0187 - accuracy: 0.9922
94/1688 [>.............................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9930
124/1688 [=>............................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9940
154/1688 [=>............................] - ETA: 2s - loss: 0.0137 - accuracy: 0.9945
184/1688 [==>...........................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9951
214/1688 [==>...........................] - ETA: 2s - loss: 0.0126 - accuracy: 0.9953
245/1688 [===>..........................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9955
276/1688 [===>..........................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9957
306/1688 [====>.........................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9956
337/1688 [====>.........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9957
367/1688 [=====>........................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9958
398/1688 [======>.......................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9959
428/1688 [======>.......................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9960
458/1688 [=======>......................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9960
488/1688 [=======>......................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9961
519/1688 [========>.....................] - ETA: 1s - loss: 0.0109 - accuracy: 0.9961
549/1688 [========>.....................] - ETA: 1s - loss: 0.0108 - accuracy: 0.9961
579/1688 [=========>....................] - ETA: 1s - loss: 0.0110 - accuracy: 0.9960
609/1688 [=========>....................] - ETA: 1s - loss: 0.0108 - accuracy: 0.9962
639/1688 [==========>...................] - ETA: 1s - loss: 0.0106 - accuracy: 0.9962
669/1688 [==========>...................] - ETA: 1s - loss: 0.0107 - accuracy: 0.9963
699/1688 [===========>..................] - ETA: 1s - loss: 0.0109 - accuracy: 0.9962
729/1688 [===========>..................] - ETA: 1s - loss: 0.0115 - accuracy: 0.9960
759/1688 [============>.................] - ETA: 1s - loss: 0.0117 - accuracy: 0.9959
789/1688 [=============>................] - ETA: 1s - loss: 0.0121 - accuracy: 0.9958
820/1688 [=============>................] - ETA: 1s - loss: 0.0126 - accuracy: 0.9956
850/1688 [==============>...............] - ETA: 1s - loss: 0.0129 - accuracy: 0.9955
880/1688 [==============>...............] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
911/1688 [===============>..............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9952
943/1688 [===============>..............] - ETA: 1s - loss: 0.0140 - accuracy: 0.9950
973/1688 [================>.............] - ETA: 1s - loss: 0.0140 - accuracy: 0.9950
1004/1688 [================>.............] - ETA: 1s - loss: 0.0143 - accuracy: 0.9948
1034/1688 [=================>............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9947
1064/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9946
1094/1688 [==================>...........] - ETA: 0s - loss: 0.0146 - accuracy: 0.9947
1124/1688 [==================>...........] - ETA: 0s - loss: 0.0146 - accuracy: 0.9946
1155/1688 [===================>..........] - ETA: 0s - loss: 0.0146 - accuracy: 0.9947
1186/1688 [====================>.........] - ETA: 0s - loss: 0.0149 - accuracy: 0.9946
1217/1688 [====================>.........] - ETA: 0s - loss: 0.0151 - accuracy: 0.9946
1247/1688 [=====================>........] - ETA: 0s - loss: 0.0152 - accuracy: 0.9945
1278/1688 [=====================>........] - ETA: 0s - loss: 0.0151 - accuracy: 0.9945
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0156 - accuracy: 0.9945
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0157 - accuracy: 0.9945
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0158 - accuracy: 0.9944
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0162 - accuracy: 0.9943
1431/1688 [========================>.....] - ETA: 0s - loss: 0.0163 - accuracy: 0.9943
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9942
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0168 - accuracy: 0.9942
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0168 - accuracy: 0.9942
1553/1688 [==========================>...] - ETA: 0s - loss: 0.0168 - accuracy: 0.9942
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0168 - accuracy: 0.9942
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0170 - accuracy: 0.9942
1643/1688 [============================>.] - ETA: 0s - loss: 0.0172 - accuracy: 0.9941
1674/1688 [============================>.] - ETA: 0s - loss: 0.0174 - accuracy: 0.9940
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0175 - accuracy: 0.9939 - val_loss: 0.0736 - val_accuracy: 0.9847
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Test accuracy: 0.9800999760627747
2. Quantize
2.1. 8-bit quantization
An Akida accelerator processes 8, 4, 2, or 1 bit integer activations and weights. Therefore, the floating point Keras model must be quantized in preparation to run on an Akida accelerator.
The QuantizeML quantize function can be used to quantize a Keras model for Akida. For this step in this example, an “8/8/8” quantization scheme will be applied to the floating point Keras model to produce 8-bit weights in the first layer, 8-bit weights in all other layers, and 8-bit activations.
The quantization process results in a Keras model with custom QuantizeML quantized layers substituted for the original Keras layers.
All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.
Note
The quantize function applies several transformations to
the original model. For example, it folds the batch normalization layers into the
corresponding neural layers. The new weights are computed according to this folding
operation.
from quantizeml.models import quantize
from quantizeml.layers import QuantizationParams
qparams = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)
model_quantized = quantize(model_keras, qparams=qparams)
model_quantized.summary()
1/1024 [..............................] - ETA: 2:30
60/1024 [>.............................] - ETA: 0s
119/1024 [==>...........................] - ETA: 0s
179/1024 [====>.........................] - ETA: 0s
237/1024 [=====>........................] - ETA: 0s
295/1024 [=======>......................] - ETA: 0s
351/1024 [=========>....................] - ETA: 0s
409/1024 [==========>...................] - ETA: 0s
466/1024 [============>.................] - ETA: 0s
525/1024 [==============>...............] - ETA: 0s
585/1024 [================>.............] - ETA: 0s
643/1024 [=================>............] - ETA: 0s
701/1024 [===================>..........] - ETA: 0s
759/1024 [=====================>........] - ETA: 0s
818/1024 [======================>.......] - ETA: 0s
876/1024 [========================>.....] - ETA: 0s
934/1024 [==========================>...] - ETA: 0s
993/1024 [============================>.] - ETA: 0s
1024/1024 [==============================] - 1s 864us/step
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
rescaling_input (InputLayer [(None, 28, 28, 1)] 0
)
rescaling (QuantizedRescali (None, 28, 28, 1) 0
ng)
conv2d (QuantizedConv2D) (None, 13, 13, 32) 320
re_lu (QuantizedReLU) (None, 13, 13, 32) 64
depthwise_conv2d (Quantized (None, 7, 7, 32) 384
DepthwiseConv2D)
conv2d_1 (QuantizedConv2D) (None, 7, 7, 64) 2112
re_lu_1 (QuantizedReLU) (None, 7, 7, 64) 128
flatten (QuantizedFlatten) (None, 3136) 0
dense (QuantizedDense) (None, 10) 31370
dequantizer (Dequantizer) [(None, 10)] 0
=================================================================
Total params: 34,378
Trainable params: 34,122
Non-trainable params: 256
_________________________________________________________________
Note
Note that the number of parameters for the floating and quantized models differs, a consequence of the BatchNormalization folding and the additional parameters added for quantization. For further details, please refer to their respective summary.
Check the quantized model accuracy.
def compile_evaluate(model):
""" Compiles and evaluates the model, then return accuracy score. """
model.compile(metrics=['accuracy'])
return model.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after 8-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 8-bit quantization: 0.9735000133514404
2.2. Effect of calibration
The previous call to quantize was made with random samples for calibration
(default parameters). While the observed drop in accuracy is minimal, that is
around 1%, it can be worse on more complex models. Therefore, it is advised to
use a set of real samples from the training set for calibration during a call
to quantize.
Note that this remains a calibration step rather than a training step in that
no output labels are required. Furthermore, any relevant data could be used for
calibration. The recommended settings for calibration that are widely used to
obtain the zoo performance are:
1024 samples
a batch size of 100
2 epochs
model_quantized = quantize(model_keras, qparams=qparams,
samples=x_train, num_samples=1024, batch_size=100, epochs=2)
1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step
1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 997us/step
Check the accuracy for the quantized and calibrated model.
print('Test accuracy after calibration:', compile_evaluate(model_quantized))
Test accuracy after calibration: 0.973800003528595
Calibrating with real samples on this model recovers the initial float accuracy.
2.3. 4-bit quantization
The accuracy of the 8/8/8 quantized model is equal to that of the Keras floating point model. In some cases, a smaller memory size for the model is required. This can be accomplished through quantization of the model to smaller bitwidths.
The model will now be quantized to 8/4/4, that is 8-bit weights in the first layer with 4-bit weights and activations in all other layers. Such a quantization scheme will usually introduce a performance drop.
qparams = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)
model_quantized = quantize(model_keras, qparams=qparams,
samples=x_train, num_samples=1024, batch_size=100, epochs=2)
1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 999us/step
1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 1ms/step
Check the 4-bit quantized accuracy.
print('Test accuracy after 4-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 4-bit quantization: 0.9652000069618225
2.4. Model fine tuning (Quantization Aware Training)
When a model suffers from an accuracy drop after quantization, fine tuning or Quantization Aware Training (QAT) may recover some or all of the original performance.
Note that since this is a fine tuning step, both the number of epochs and learning rate are expected to be lower than during the initial float training.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=Adam(learning_rate=1e-4),
metrics=['accuracy'])
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
Epoch 1/5
1/1688 [..............................] - ETA: 59:41 - loss: 0.0235 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0846 - accuracy: 0.9663
25/1688 [..............................] - ETA: 7s - loss: 0.0768 - accuracy: 0.9712
37/1688 [..............................] - ETA: 7s - loss: 0.0635 - accuracy: 0.9780
49/1688 [..............................] - ETA: 7s - loss: 0.0551 - accuracy: 0.9815
61/1688 [>.............................] - ETA: 7s - loss: 0.0497 - accuracy: 0.9836
73/1688 [>.............................] - ETA: 7s - loss: 0.0499 - accuracy: 0.9842
84/1688 [>.............................] - ETA: 7s - loss: 0.0477 - accuracy: 0.9844
96/1688 [>.............................] - ETA: 7s - loss: 0.0457 - accuracy: 0.9854
108/1688 [>.............................] - ETA: 7s - loss: 0.0449 - accuracy: 0.9850
120/1688 [=>............................] - ETA: 6s - loss: 0.0449 - accuracy: 0.9849
132/1688 [=>............................] - ETA: 6s - loss: 0.0436 - accuracy: 0.9853
144/1688 [=>............................] - ETA: 6s - loss: 0.0417 - accuracy: 0.9859
156/1688 [=>............................] - ETA: 6s - loss: 0.0409 - accuracy: 0.9860
167/1688 [=>............................] - ETA: 6s - loss: 0.0396 - accuracy: 0.9865
179/1688 [==>...........................] - ETA: 6s - loss: 0.0393 - accuracy: 0.9864
191/1688 [==>...........................] - ETA: 6s - loss: 0.0389 - accuracy: 0.9863
203/1688 [==>...........................] - ETA: 6s - loss: 0.0384 - accuracy: 0.9866
215/1688 [==>...........................] - ETA: 6s - loss: 0.0388 - accuracy: 0.9868
227/1688 [===>..........................] - ETA: 6s - loss: 0.0381 - accuracy: 0.9869
239/1688 [===>..........................] - ETA: 6s - loss: 0.0372 - accuracy: 0.9872
251/1688 [===>..........................] - ETA: 6s - loss: 0.0370 - accuracy: 0.9874
263/1688 [===>..........................] - ETA: 6s - loss: 0.0366 - accuracy: 0.9875
275/1688 [===>..........................] - ETA: 6s - loss: 0.0357 - accuracy: 0.9878
287/1688 [====>.........................] - ETA: 6s - loss: 0.0356 - accuracy: 0.9880
299/1688 [====>.........................] - ETA: 6s - loss: 0.0349 - accuracy: 0.9883
311/1688 [====>.........................] - ETA: 6s - loss: 0.0343 - accuracy: 0.9884
323/1688 [====>.........................] - ETA: 6s - loss: 0.0340 - accuracy: 0.9885
335/1688 [====>.........................] - ETA: 6s - loss: 0.0336 - accuracy: 0.9887
347/1688 [=====>........................] - ETA: 5s - loss: 0.0334 - accuracy: 0.9888
359/1688 [=====>........................] - ETA: 5s - loss: 0.0329 - accuracy: 0.9889
371/1688 [=====>........................] - ETA: 5s - loss: 0.0323 - accuracy: 0.9893
383/1688 [=====>........................] - ETA: 5s - loss: 0.0317 - accuracy: 0.9896
395/1688 [======>.......................] - ETA: 5s - loss: 0.0313 - accuracy: 0.9897
407/1688 [======>.......................] - ETA: 5s - loss: 0.0309 - accuracy: 0.9899
419/1688 [======>.......................] - ETA: 5s - loss: 0.0306 - accuracy: 0.9899
431/1688 [======>.......................] - ETA: 5s - loss: 0.0305 - accuracy: 0.9900
443/1688 [======>.......................] - ETA: 5s - loss: 0.0301 - accuracy: 0.9902
455/1688 [=======>......................] - ETA: 5s - loss: 0.0297 - accuracy: 0.9904
467/1688 [=======>......................] - ETA: 5s - loss: 0.0293 - accuracy: 0.9906
479/1688 [=======>......................] - ETA: 5s - loss: 0.0289 - accuracy: 0.9907
491/1688 [=======>......................] - ETA: 5s - loss: 0.0291 - accuracy: 0.9906
503/1688 [=======>......................] - ETA: 5s - loss: 0.0288 - accuracy: 0.9908
515/1688 [========>.....................] - ETA: 5s - loss: 0.0284 - accuracy: 0.9910
527/1688 [========>.....................] - ETA: 5s - loss: 0.0282 - accuracy: 0.9911
539/1688 [========>.....................] - ETA: 5s - loss: 0.0278 - accuracy: 0.9913
551/1688 [========>.....................] - ETA: 5s - loss: 0.0276 - accuracy: 0.9914
563/1688 [=========>....................] - ETA: 5s - loss: 0.0274 - accuracy: 0.9915
575/1688 [=========>....................] - ETA: 4s - loss: 0.0272 - accuracy: 0.9916
587/1688 [=========>....................] - ETA: 4s - loss: 0.0270 - accuracy: 0.9917
599/1688 [=========>....................] - ETA: 4s - loss: 0.0267 - accuracy: 0.9918
611/1688 [=========>....................] - ETA: 4s - loss: 0.0266 - accuracy: 0.9919
623/1688 [==========>...................] - ETA: 4s - loss: 0.0265 - accuracy: 0.9919
635/1688 [==========>...................] - ETA: 4s - loss: 0.0265 - accuracy: 0.9919
647/1688 [==========>...................] - ETA: 4s - loss: 0.0263 - accuracy: 0.9920
659/1688 [==========>...................] - ETA: 4s - loss: 0.0262 - accuracy: 0.9920
671/1688 [==========>...................] - ETA: 4s - loss: 0.0261 - accuracy: 0.9920
683/1688 [===========>..................] - ETA: 4s - loss: 0.0258 - accuracy: 0.9921
695/1688 [===========>..................] - ETA: 4s - loss: 0.0258 - accuracy: 0.9921
707/1688 [===========>..................] - ETA: 4s - loss: 0.0256 - accuracy: 0.9922
719/1688 [===========>..................] - ETA: 4s - loss: 0.0254 - accuracy: 0.9923
731/1688 [===========>..................] - ETA: 4s - loss: 0.0252 - accuracy: 0.9923
743/1688 [============>.................] - ETA: 4s - loss: 0.0251 - accuracy: 0.9923
755/1688 [============>.................] - ETA: 4s - loss: 0.0249 - accuracy: 0.9924
767/1688 [============>.................] - ETA: 4s - loss: 0.0247 - accuracy: 0.9925
779/1688 [============>.................] - ETA: 4s - loss: 0.0245 - accuracy: 0.9926
791/1688 [=============>................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9926
803/1688 [=============>................] - ETA: 3s - loss: 0.0245 - accuracy: 0.9926
815/1688 [=============>................] - ETA: 3s - loss: 0.0242 - accuracy: 0.9927
827/1688 [=============>................] - ETA: 3s - loss: 0.0241 - accuracy: 0.9927
839/1688 [=============>................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9928
851/1688 [==============>...............] - ETA: 3s - loss: 0.0237 - accuracy: 0.9929
863/1688 [==============>...............] - ETA: 3s - loss: 0.0235 - accuracy: 0.9930
875/1688 [==============>...............] - ETA: 3s - loss: 0.0233 - accuracy: 0.9931
887/1688 [==============>...............] - ETA: 3s - loss: 0.0232 - accuracy: 0.9931
899/1688 [==============>...............] - ETA: 3s - loss: 0.0232 - accuracy: 0.9930
911/1688 [===============>..............] - ETA: 3s - loss: 0.0231 - accuracy: 0.9931
923/1688 [===============>..............] - ETA: 3s - loss: 0.0230 - accuracy: 0.9931
935/1688 [===============>..............] - ETA: 3s - loss: 0.0230 - accuracy: 0.9931
947/1688 [===============>..............] - ETA: 3s - loss: 0.0229 - accuracy: 0.9931
959/1688 [================>.............] - ETA: 3s - loss: 0.0228 - accuracy: 0.9931
971/1688 [================>.............] - ETA: 3s - loss: 0.0228 - accuracy: 0.9932
983/1688 [================>.............] - ETA: 3s - loss: 0.0227 - accuracy: 0.9932
995/1688 [================>.............] - ETA: 3s - loss: 0.0226 - accuracy: 0.9933
1007/1688 [================>.............] - ETA: 3s - loss: 0.0224 - accuracy: 0.9934
1019/1688 [=================>............] - ETA: 2s - loss: 0.0224 - accuracy: 0.9934
1031/1688 [=================>............] - ETA: 2s - loss: 0.0225 - accuracy: 0.9934
1043/1688 [=================>............] - ETA: 2s - loss: 0.0226 - accuracy: 0.9933
1055/1688 [=================>............] - ETA: 2s - loss: 0.0225 - accuracy: 0.9933
1067/1688 [=================>............] - ETA: 2s - loss: 0.0224 - accuracy: 0.9934
1079/1688 [==================>...........] - ETA: 2s - loss: 0.0223 - accuracy: 0.9935
1091/1688 [==================>...........] - ETA: 2s - loss: 0.0222 - accuracy: 0.9935
1103/1688 [==================>...........] - ETA: 2s - loss: 0.0222 - accuracy: 0.9935
1115/1688 [==================>...........] - ETA: 2s - loss: 0.0223 - accuracy: 0.9935
1127/1688 [===================>..........] - ETA: 2s - loss: 0.0222 - accuracy: 0.9935
1139/1688 [===================>..........] - ETA: 2s - loss: 0.0222 - accuracy: 0.9934
1151/1688 [===================>..........] - ETA: 2s - loss: 0.0221 - accuracy: 0.9935
1163/1688 [===================>..........] - ETA: 2s - loss: 0.0220 - accuracy: 0.9935
1175/1688 [===================>..........] - ETA: 2s - loss: 0.0220 - accuracy: 0.9935
1187/1688 [====================>.........] - ETA: 2s - loss: 0.0220 - accuracy: 0.9934
1199/1688 [====================>.........] - ETA: 2s - loss: 0.0219 - accuracy: 0.9935
1211/1688 [====================>.........] - ETA: 2s - loss: 0.0219 - accuracy: 0.9934
1223/1688 [====================>.........] - ETA: 2s - loss: 0.0219 - accuracy: 0.9934
1235/1688 [====================>.........] - ETA: 2s - loss: 0.0219 - accuracy: 0.9934
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0218 - accuracy: 0.9934
1259/1688 [=====================>........] - ETA: 1s - loss: 0.0218 - accuracy: 0.9933
1271/1688 [=====================>........] - ETA: 1s - loss: 0.0218 - accuracy: 0.9933
1283/1688 [=====================>........] - ETA: 1s - loss: 0.0218 - accuracy: 0.9933
1295/1688 [======================>.......] - ETA: 1s - loss: 0.0217 - accuracy: 0.9933
1307/1688 [======================>.......] - ETA: 1s - loss: 0.0216 - accuracy: 0.9934
1319/1688 [======================>.......] - ETA: 1s - loss: 0.0216 - accuracy: 0.9934
1331/1688 [======================>.......] - ETA: 1s - loss: 0.0215 - accuracy: 0.9934
1343/1688 [======================>.......] - ETA: 1s - loss: 0.0215 - accuracy: 0.9934
1355/1688 [=======================>......] - ETA: 1s - loss: 0.0214 - accuracy: 0.9934
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0214 - accuracy: 0.9934
1379/1688 [=======================>......] - ETA: 1s - loss: 0.0214 - accuracy: 0.9934
1391/1688 [=======================>......] - ETA: 1s - loss: 0.0214 - accuracy: 0.9934
1403/1688 [=======================>......] - ETA: 1s - loss: 0.0214 - accuracy: 0.9934
1415/1688 [========================>.....] - ETA: 1s - loss: 0.0213 - accuracy: 0.9934
1427/1688 [========================>.....] - ETA: 1s - loss: 0.0213 - accuracy: 0.9934
1439/1688 [========================>.....] - ETA: 1s - loss: 0.0212 - accuracy: 0.9934
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0212 - accuracy: 0.9934
1463/1688 [=========================>....] - ETA: 1s - loss: 0.0211 - accuracy: 0.9935
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0211 - accuracy: 0.9935
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0210 - accuracy: 0.9935
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0210 - accuracy: 0.9935
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0209 - accuracy: 0.9935
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0208 - accuracy: 0.9936
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0208 - accuracy: 0.9936
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0208 - accuracy: 0.9936
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0208 - accuracy: 0.9936
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0208 - accuracy: 0.9936
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0207 - accuracy: 0.9936
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0207 - accuracy: 0.9937
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0206 - accuracy: 0.9937
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0205 - accuracy: 0.9937
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0205 - accuracy: 0.9937
1643/1688 [============================>.] - ETA: 0s - loss: 0.0204 - accuracy: 0.9937
1655/1688 [============================>.] - ETA: 0s - loss: 0.0204 - accuracy: 0.9937
1667/1688 [============================>.] - ETA: 0s - loss: 0.0204 - accuracy: 0.9937
1679/1688 [============================>.] - ETA: 0s - loss: 0.0203 - accuracy: 0.9938
1688/1688 [==============================] - 11s 5ms/step - loss: 0.0203 - accuracy: 0.9938 - val_loss: 0.0513 - val_accuracy: 0.9865
Epoch 2/5
1/1688 [..............................] - ETA: 7s - loss: 0.0016 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0087 - accuracy: 0.9952
25/1688 [..............................] - ETA: 7s - loss: 0.0136 - accuracy: 0.9937
37/1688 [..............................] - ETA: 7s - loss: 0.0114 - accuracy: 0.9958
49/1688 [..............................] - ETA: 7s - loss: 0.0118 - accuracy: 0.9968
61/1688 [>.............................] - ETA: 7s - loss: 0.0148 - accuracy: 0.9959
73/1688 [>.............................] - ETA: 7s - loss: 0.0157 - accuracy: 0.9957
85/1688 [>.............................] - ETA: 7s - loss: 0.0154 - accuracy: 0.9956
97/1688 [>.............................] - ETA: 7s - loss: 0.0150 - accuracy: 0.9958
109/1688 [>.............................] - ETA: 7s - loss: 0.0152 - accuracy: 0.9957
121/1688 [=>............................] - ETA: 6s - loss: 0.0152 - accuracy: 0.9956
133/1688 [=>............................] - ETA: 6s - loss: 0.0154 - accuracy: 0.9955
145/1688 [=>............................] - ETA: 6s - loss: 0.0149 - accuracy: 0.9957
157/1688 [=>............................] - ETA: 6s - loss: 0.0148 - accuracy: 0.9956
169/1688 [==>...........................] - ETA: 6s - loss: 0.0157 - accuracy: 0.9954
181/1688 [==>...........................] - ETA: 6s - loss: 0.0161 - accuracy: 0.9953
193/1688 [==>...........................] - ETA: 6s - loss: 0.0157 - accuracy: 0.9955
205/1688 [==>...........................] - ETA: 6s - loss: 0.0151 - accuracy: 0.9957
217/1688 [==>...........................] - ETA: 6s - loss: 0.0147 - accuracy: 0.9960
229/1688 [===>..........................] - ETA: 6s - loss: 0.0144 - accuracy: 0.9962
241/1688 [===>..........................] - ETA: 6s - loss: 0.0149 - accuracy: 0.9957
253/1688 [===>..........................] - ETA: 6s - loss: 0.0146 - accuracy: 0.9958
265/1688 [===>..........................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9960
277/1688 [===>..........................] - ETA: 6s - loss: 0.0145 - accuracy: 0.9958
289/1688 [====>.........................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9960
301/1688 [====>.........................] - ETA: 6s - loss: 0.0141 - accuracy: 0.9961
313/1688 [====>.........................] - ETA: 6s - loss: 0.0141 - accuracy: 0.9961
325/1688 [====>.........................] - ETA: 6s - loss: 0.0141 - accuracy: 0.9962
337/1688 [====>.........................] - ETA: 6s - loss: 0.0140 - accuracy: 0.9961
349/1688 [=====>........................] - ETA: 5s - loss: 0.0140 - accuracy: 0.9961
360/1688 [=====>........................] - ETA: 5s - loss: 0.0141 - accuracy: 0.9962
372/1688 [=====>........................] - ETA: 5s - loss: 0.0139 - accuracy: 0.9962
384/1688 [=====>........................] - ETA: 5s - loss: 0.0140 - accuracy: 0.9961
396/1688 [======>.......................] - ETA: 5s - loss: 0.0144 - accuracy: 0.9960
408/1688 [======>.......................] - ETA: 5s - loss: 0.0142 - accuracy: 0.9961
420/1688 [======>.......................] - ETA: 5s - loss: 0.0142 - accuracy: 0.9961
432/1688 [======>.......................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9961
443/1688 [======>.......................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9959
455/1688 [=======>......................] - ETA: 5s - loss: 0.0142 - accuracy: 0.9960
467/1688 [=======>......................] - ETA: 5s - loss: 0.0140 - accuracy: 0.9961
479/1688 [=======>......................] - ETA: 5s - loss: 0.0139 - accuracy: 0.9962
491/1688 [=======>......................] - ETA: 5s - loss: 0.0138 - accuracy: 0.9961
503/1688 [=======>......................] - ETA: 5s - loss: 0.0136 - accuracy: 0.9962
515/1688 [========>.....................] - ETA: 5s - loss: 0.0135 - accuracy: 0.9962
527/1688 [========>.....................] - ETA: 5s - loss: 0.0136 - accuracy: 0.9962
538/1688 [========>.....................] - ETA: 5s - loss: 0.0135 - accuracy: 0.9962
550/1688 [========>.....................] - ETA: 5s - loss: 0.0134 - accuracy: 0.9962
562/1688 [========>.....................] - ETA: 5s - loss: 0.0135 - accuracy: 0.9961
574/1688 [=========>....................] - ETA: 4s - loss: 0.0137 - accuracy: 0.9960
586/1688 [=========>....................] - ETA: 4s - loss: 0.0137 - accuracy: 0.9961
598/1688 [=========>....................] - ETA: 4s - loss: 0.0136 - accuracy: 0.9961
610/1688 [=========>....................] - ETA: 4s - loss: 0.0136 - accuracy: 0.9961
622/1688 [==========>...................] - ETA: 4s - loss: 0.0137 - accuracy: 0.9960
634/1688 [==========>...................] - ETA: 4s - loss: 0.0138 - accuracy: 0.9961
646/1688 [==========>...................] - ETA: 4s - loss: 0.0137 - accuracy: 0.9961
658/1688 [==========>...................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9960
670/1688 [==========>...................] - ETA: 4s - loss: 0.0141 - accuracy: 0.9959
682/1688 [===========>..................] - ETA: 4s - loss: 0.0140 - accuracy: 0.9959
694/1688 [===========>..................] - ETA: 4s - loss: 0.0140 - accuracy: 0.9959
706/1688 [===========>..................] - ETA: 4s - loss: 0.0140 - accuracy: 0.9959
718/1688 [===========>..................] - ETA: 4s - loss: 0.0141 - accuracy: 0.9959
730/1688 [===========>..................] - ETA: 4s - loss: 0.0141 - accuracy: 0.9959
742/1688 [============>.................] - ETA: 4s - loss: 0.0141 - accuracy: 0.9959
754/1688 [============>.................] - ETA: 4s - loss: 0.0140 - accuracy: 0.9960
766/1688 [============>.................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9960
778/1688 [============>.................] - ETA: 4s - loss: 0.0138 - accuracy: 0.9961
790/1688 [=============>................] - ETA: 4s - loss: 0.0138 - accuracy: 0.9961
802/1688 [=============>................] - ETA: 3s - loss: 0.0137 - accuracy: 0.9961
814/1688 [=============>................] - ETA: 3s - loss: 0.0137 - accuracy: 0.9962
826/1688 [=============>................] - ETA: 3s - loss: 0.0138 - accuracy: 0.9961
837/1688 [=============>................] - ETA: 3s - loss: 0.0138 - accuracy: 0.9961
849/1688 [==============>...............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
861/1688 [==============>...............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
873/1688 [==============>...............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9961
885/1688 [==============>...............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9962
897/1688 [==============>...............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
909/1688 [===============>..............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9960
921/1688 [===============>..............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
933/1688 [===============>..............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
945/1688 [===============>..............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
957/1688 [================>.............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9961
968/1688 [================>.............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9961
980/1688 [================>.............] - ETA: 3s - loss: 0.0137 - accuracy: 0.9962
991/1688 [================>.............] - ETA: 3s - loss: 0.0137 - accuracy: 0.9961
1003/1688 [================>.............] - ETA: 3s - loss: 0.0138 - accuracy: 0.9961
1015/1688 [=================>............] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
1027/1688 [=================>............] - ETA: 2s - loss: 0.0139 - accuracy: 0.9961
1039/1688 [=================>............] - ETA: 2s - loss: 0.0139 - accuracy: 0.9961
1051/1688 [=================>............] - ETA: 2s - loss: 0.0138 - accuracy: 0.9961
1063/1688 [=================>............] - ETA: 2s - loss: 0.0137 - accuracy: 0.9961
1075/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1087/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1099/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1111/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1123/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1135/1688 [===================>..........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1147/1688 [===================>..........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9962
1159/1688 [===================>..........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9962
1171/1688 [===================>..........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9962
1183/1688 [====================>.........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1195/1688 [====================>.........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9962
1207/1688 [====================>.........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9962
1219/1688 [====================>.........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9962
1231/1688 [====================>.........] - ETA: 2s - loss: 0.0138 - accuracy: 0.9962
1243/1688 [=====================>........] - ETA: 1s - loss: 0.0138 - accuracy: 0.9962
1255/1688 [=====================>........] - ETA: 1s - loss: 0.0137 - accuracy: 0.9962
1267/1688 [=====================>........] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1279/1688 [=====================>........] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1291/1688 [=====================>........] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1303/1688 [======================>.......] - ETA: 1s - loss: 0.0137 - accuracy: 0.9962
1315/1688 [======================>.......] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1327/1688 [======================>.......] - ETA: 1s - loss: 0.0137 - accuracy: 0.9962
1339/1688 [======================>.......] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1351/1688 [=======================>......] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1363/1688 [=======================>......] - ETA: 1s - loss: 0.0135 - accuracy: 0.9962
1375/1688 [=======================>......] - ETA: 1s - loss: 0.0135 - accuracy: 0.9962
1387/1688 [=======================>......] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1399/1688 [=======================>......] - ETA: 1s - loss: 0.0135 - accuracy: 0.9962
1411/1688 [========================>.....] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1423/1688 [========================>.....] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1435/1688 [========================>.....] - ETA: 1s - loss: 0.0137 - accuracy: 0.9962
1447/1688 [========================>.....] - ETA: 1s - loss: 0.0136 - accuracy: 0.9962
1459/1688 [========================>.....] - ETA: 1s - loss: 0.0137 - accuracy: 0.9962
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0137 - accuracy: 0.9962
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0138 - accuracy: 0.9961
1494/1688 [=========================>....] - ETA: 0s - loss: 0.0137 - accuracy: 0.9962
1506/1688 [=========================>....] - ETA: 0s - loss: 0.0138 - accuracy: 0.9962
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0137 - accuracy: 0.9962
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0137 - accuracy: 0.9962
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1554/1688 [==========================>...] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0139 - accuracy: 0.9963
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1638/1688 [============================>.] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1650/1688 [============================>.] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1662/1688 [============================>.] - ETA: 0s - loss: 0.0139 - accuracy: 0.9962
1674/1688 [============================>.] - ETA: 0s - loss: 0.0140 - accuracy: 0.9961
1686/1688 [============================>.] - ETA: 0s - loss: 0.0140 - accuracy: 0.9961
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0140 - accuracy: 0.9961 - val_loss: 0.0490 - val_accuracy: 0.9863
Epoch 3/5
1/1688 [..............................] - ETA: 7s - loss: 0.0032 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0135 - accuracy: 0.9952
25/1688 [..............................] - ETA: 7s - loss: 0.0104 - accuracy: 0.9975
37/1688 [..............................] - ETA: 7s - loss: 0.0133 - accuracy: 0.9958
49/1688 [..............................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9968
61/1688 [>.............................] - ETA: 7s - loss: 0.0115 - accuracy: 0.9974
73/1688 [>.............................] - ETA: 7s - loss: 0.0115 - accuracy: 0.9966
85/1688 [>.............................] - ETA: 7s - loss: 0.0107 - accuracy: 0.9971
97/1688 [>.............................] - ETA: 7s - loss: 0.0105 - accuracy: 0.9971
109/1688 [>.............................] - ETA: 7s - loss: 0.0112 - accuracy: 0.9968
121/1688 [=>............................] - ETA: 6s - loss: 0.0108 - accuracy: 0.9972
133/1688 [=>............................] - ETA: 6s - loss: 0.0109 - accuracy: 0.9972
145/1688 [=>............................] - ETA: 6s - loss: 0.0113 - accuracy: 0.9972
157/1688 [=>............................] - ETA: 6s - loss: 0.0111 - accuracy: 0.9972
169/1688 [==>...........................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9970
181/1688 [==>...........................] - ETA: 6s - loss: 0.0112 - accuracy: 0.9972
193/1688 [==>...........................] - ETA: 6s - loss: 0.0109 - accuracy: 0.9974
205/1688 [==>...........................] - ETA: 6s - loss: 0.0113 - accuracy: 0.9973
217/1688 [==>...........................] - ETA: 6s - loss: 0.0115 - accuracy: 0.9968
229/1688 [===>..........................] - ETA: 6s - loss: 0.0116 - accuracy: 0.9969
241/1688 [===>..........................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9968
253/1688 [===>..........................] - ETA: 6s - loss: 0.0116 - accuracy: 0.9968
265/1688 [===>..........................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9967
277/1688 [===>..........................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9967
289/1688 [====>.........................] - ETA: 6s - loss: 0.0119 - accuracy: 0.9965
301/1688 [====>.........................] - ETA: 6s - loss: 0.0119 - accuracy: 0.9966
313/1688 [====>.........................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9967
325/1688 [====>.........................] - ETA: 6s - loss: 0.0115 - accuracy: 0.9967
337/1688 [====>.........................] - ETA: 6s - loss: 0.0115 - accuracy: 0.9967
349/1688 [=====>........................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9967
361/1688 [=====>........................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9968
373/1688 [=====>........................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9969
384/1688 [=====>........................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9970
396/1688 [======>.......................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9970
408/1688 [======>.......................] - ETA: 5s - loss: 0.0110 - accuracy: 0.9971
420/1688 [======>.......................] - ETA: 5s - loss: 0.0110 - accuracy: 0.9971
432/1688 [======>.......................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9970
444/1688 [======>.......................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9971
456/1688 [=======>......................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9971
468/1688 [=======>......................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9972
480/1688 [=======>......................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9973
491/1688 [=======>......................] - ETA: 5s - loss: 0.0108 - accuracy: 0.9973
503/1688 [=======>......................] - ETA: 5s - loss: 0.0107 - accuracy: 0.9973
515/1688 [========>.....................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9972
527/1688 [========>.....................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9970
539/1688 [========>.....................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9970
551/1688 [========>.....................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9969
563/1688 [=========>....................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9969
575/1688 [=========>....................] - ETA: 4s - loss: 0.0112 - accuracy: 0.9969
587/1688 [=========>....................] - ETA: 4s - loss: 0.0113 - accuracy: 0.9969
599/1688 [=========>....................] - ETA: 4s - loss: 0.0112 - accuracy: 0.9969
611/1688 [=========>....................] - ETA: 4s - loss: 0.0112 - accuracy: 0.9969
623/1688 [==========>...................] - ETA: 4s - loss: 0.0112 - accuracy: 0.9970
635/1688 [==========>...................] - ETA: 4s - loss: 0.0111 - accuracy: 0.9970
647/1688 [==========>...................] - ETA: 4s - loss: 0.0111 - accuracy: 0.9971
659/1688 [==========>...................] - ETA: 4s - loss: 0.0112 - accuracy: 0.9971
671/1688 [==========>...................] - ETA: 4s - loss: 0.0112 - accuracy: 0.9971
683/1688 [===========>..................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9971
695/1688 [===========>..................] - ETA: 4s - loss: 0.0113 - accuracy: 0.9970
707/1688 [===========>..................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9970
719/1688 [===========>..................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9971
731/1688 [===========>..................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9971
743/1688 [============>.................] - ETA: 4s - loss: 0.0115 - accuracy: 0.9971
755/1688 [============>.................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9971
767/1688 [============>.................] - ETA: 4s - loss: 0.0114 - accuracy: 0.9970
779/1688 [============>.................] - ETA: 4s - loss: 0.0115 - accuracy: 0.9970
791/1688 [=============>................] - ETA: 3s - loss: 0.0114 - accuracy: 0.9970
802/1688 [=============>................] - ETA: 3s - loss: 0.0115 - accuracy: 0.9970
814/1688 [=============>................] - ETA: 3s - loss: 0.0114 - accuracy: 0.9970
826/1688 [=============>................] - ETA: 3s - loss: 0.0114 - accuracy: 0.9971
838/1688 [=============>................] - ETA: 3s - loss: 0.0114 - accuracy: 0.9971
850/1688 [==============>...............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9971
862/1688 [==============>...............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9971
874/1688 [==============>...............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9971
886/1688 [==============>...............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9970
898/1688 [==============>...............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9969
910/1688 [===============>..............] - ETA: 3s - loss: 0.0118 - accuracy: 0.9968
921/1688 [===============>..............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9969
933/1688 [===============>..............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9969
945/1688 [===============>..............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9969
957/1688 [================>.............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9969
969/1688 [================>.............] - ETA: 3s - loss: 0.0118 - accuracy: 0.9968
981/1688 [================>.............] - ETA: 3s - loss: 0.0118 - accuracy: 0.9969
993/1688 [================>.............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9969
1005/1688 [================>.............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9969
1017/1688 [=================>............] - ETA: 2s - loss: 0.0118 - accuracy: 0.9968
1029/1688 [=================>............] - ETA: 2s - loss: 0.0118 - accuracy: 0.9968
1041/1688 [=================>............] - ETA: 2s - loss: 0.0119 - accuracy: 0.9968
1053/1688 [=================>............] - ETA: 2s - loss: 0.0119 - accuracy: 0.9968
1065/1688 [=================>............] - ETA: 2s - loss: 0.0119 - accuracy: 0.9968
1077/1688 [==================>...........] - ETA: 2s - loss: 0.0118 - accuracy: 0.9968
1089/1688 [==================>...........] - ETA: 2s - loss: 0.0119 - accuracy: 0.9968
1101/1688 [==================>...........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9967
1113/1688 [==================>...........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9967
1125/1688 [==================>...........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9967
1137/1688 [===================>..........] - ETA: 2s - loss: 0.0119 - accuracy: 0.9968
1149/1688 [===================>..........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9967
1161/1688 [===================>..........] - ETA: 2s - loss: 0.0121 - accuracy: 0.9967
1173/1688 [===================>..........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9967
1185/1688 [====================>.........] - ETA: 2s - loss: 0.0121 - accuracy: 0.9967
1197/1688 [====================>.........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9967
1209/1688 [====================>.........] - ETA: 2s - loss: 0.0121 - accuracy: 0.9967
1221/1688 [====================>.........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9967
1233/1688 [====================>.........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9968
1245/1688 [=====================>........] - ETA: 1s - loss: 0.0120 - accuracy: 0.9968
1257/1688 [=====================>........] - ETA: 1s - loss: 0.0120 - accuracy: 0.9968
1269/1688 [=====================>........] - ETA: 1s - loss: 0.0119 - accuracy: 0.9968
1281/1688 [=====================>........] - ETA: 1s - loss: 0.0120 - accuracy: 0.9968
1293/1688 [=====================>........] - ETA: 1s - loss: 0.0120 - accuracy: 0.9968
1305/1688 [======================>.......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9968
1317/1688 [======================>.......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9968
1329/1688 [======================>.......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9967
1341/1688 [======================>.......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9968
1353/1688 [=======================>......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9967
1365/1688 [=======================>......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9967
1377/1688 [=======================>......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9968
1389/1688 [=======================>......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9968
1401/1688 [=======================>......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9967
1413/1688 [========================>.....] - ETA: 1s - loss: 0.0120 - accuracy: 0.9967
1425/1688 [========================>.....] - ETA: 1s - loss: 0.0120 - accuracy: 0.9968
1437/1688 [========================>.....] - ETA: 1s - loss: 0.0120 - accuracy: 0.9968
1449/1688 [========================>.....] - ETA: 1s - loss: 0.0119 - accuracy: 0.9968
1461/1688 [========================>.....] - ETA: 1s - loss: 0.0119 - accuracy: 0.9969
1473/1688 [=========================>....] - ETA: 0s - loss: 0.0120 - accuracy: 0.9968
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0120 - accuracy: 0.9968
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0120 - accuracy: 0.9968
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0121 - accuracy: 0.9967
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0121 - accuracy: 0.9968
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0122 - accuracy: 0.9967
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0122 - accuracy: 0.9967
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0123 - accuracy: 0.9967
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0123 - accuracy: 0.9967
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0123 - accuracy: 0.9967
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0123 - accuracy: 0.9967
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0123 - accuracy: 0.9967
1617/1688 [===========================>..] - ETA: 0s - loss: 0.0122 - accuracy: 0.9967
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0122 - accuracy: 0.9967
1641/1688 [============================>.] - ETA: 0s - loss: 0.0121 - accuracy: 0.9967
1653/1688 [============================>.] - ETA: 0s - loss: 0.0122 - accuracy: 0.9967
1665/1688 [============================>.] - ETA: 0s - loss: 0.0122 - accuracy: 0.9967
1677/1688 [============================>.] - ETA: 0s - loss: 0.0122 - accuracy: 0.9967
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0121 - accuracy: 0.9967 - val_loss: 0.0494 - val_accuracy: 0.9865
Epoch 4/5
1/1688 [..............................] - ETA: 7s - loss: 0.0017 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0044 - accuracy: 1.0000
25/1688 [..............................] - ETA: 7s - loss: 0.0055 - accuracy: 1.0000
37/1688 [..............................] - ETA: 7s - loss: 0.0055 - accuracy: 0.9992
49/1688 [..............................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9994
61/1688 [>.............................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9995
73/1688 [>.............................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9996
85/1688 [>.............................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9996
97/1688 [>.............................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9997
108/1688 [>.............................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9994
119/1688 [=>............................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9992
131/1688 [=>............................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9993
143/1688 [=>............................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9993
155/1688 [=>............................] - ETA: 6s - loss: 0.0074 - accuracy: 0.9990
167/1688 [=>............................] - ETA: 6s - loss: 0.0071 - accuracy: 0.9991
179/1688 [==>...........................] - ETA: 6s - loss: 0.0072 - accuracy: 0.9988
190/1688 [==>...........................] - ETA: 6s - loss: 0.0072 - accuracy: 0.9988
202/1688 [==>...........................] - ETA: 6s - loss: 0.0070 - accuracy: 0.9989
214/1688 [==>...........................] - ETA: 6s - loss: 0.0078 - accuracy: 0.9985
226/1688 [===>..........................] - ETA: 6s - loss: 0.0081 - accuracy: 0.9985
238/1688 [===>..........................] - ETA: 6s - loss: 0.0083 - accuracy: 0.9984
250/1688 [===>..........................] - ETA: 6s - loss: 0.0084 - accuracy: 0.9985
262/1688 [===>..........................] - ETA: 6s - loss: 0.0085 - accuracy: 0.9984
274/1688 [===>..........................] - ETA: 6s - loss: 0.0085 - accuracy: 0.9984
286/1688 [====>.........................] - ETA: 6s - loss: 0.0086 - accuracy: 0.9983
298/1688 [====>.........................] - ETA: 6s - loss: 0.0086 - accuracy: 0.9982
310/1688 [====>.........................] - ETA: 6s - loss: 0.0085 - accuracy: 0.9983
321/1688 [====>.........................] - ETA: 6s - loss: 0.0086 - accuracy: 0.9982
333/1688 [====>.........................] - ETA: 6s - loss: 0.0087 - accuracy: 0.9982
345/1688 [=====>........................] - ETA: 6s - loss: 0.0086 - accuracy: 0.9983
357/1688 [=====>........................] - ETA: 5s - loss: 0.0088 - accuracy: 0.9983
369/1688 [=====>........................] - ETA: 5s - loss: 0.0089 - accuracy: 0.9983
381/1688 [=====>........................] - ETA: 5s - loss: 0.0089 - accuracy: 0.9984
393/1688 [=====>........................] - ETA: 5s - loss: 0.0089 - accuracy: 0.9984
405/1688 [======>.......................] - ETA: 5s - loss: 0.0088 - accuracy: 0.9985
417/1688 [======>.......................] - ETA: 5s - loss: 0.0088 - accuracy: 0.9984
429/1688 [======>.......................] - ETA: 5s - loss: 0.0088 - accuracy: 0.9985
441/1688 [======>.......................] - ETA: 5s - loss: 0.0087 - accuracy: 0.9985
453/1688 [=======>......................] - ETA: 5s - loss: 0.0087 - accuracy: 0.9985
465/1688 [=======>......................] - ETA: 5s - loss: 0.0088 - accuracy: 0.9984
477/1688 [=======>......................] - ETA: 5s - loss: 0.0090 - accuracy: 0.9982
489/1688 [=======>......................] - ETA: 5s - loss: 0.0091 - accuracy: 0.9982
501/1688 [=======>......................] - ETA: 5s - loss: 0.0090 - accuracy: 0.9983
513/1688 [========>.....................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9982
524/1688 [========>.....................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9982
536/1688 [========>.....................] - ETA: 5s - loss: 0.0091 - accuracy: 0.9983
548/1688 [========>.....................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9982
560/1688 [========>.....................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9982
571/1688 [=========>....................] - ETA: 4s - loss: 0.0095 - accuracy: 0.9981
583/1688 [=========>....................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
595/1688 [=========>....................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9980
607/1688 [=========>....................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
619/1688 [==========>...................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
631/1688 [==========>...................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9979
643/1688 [==========>...................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9979
655/1688 [==========>...................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9979
667/1688 [==========>...................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9979
679/1688 [===========>..................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
691/1688 [===========>..................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
703/1688 [===========>..................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
715/1688 [===========>..................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
727/1688 [===========>..................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
739/1688 [============>.................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
751/1688 [============>.................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
763/1688 [============>.................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9979
775/1688 [============>.................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9979
787/1688 [============>.................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
799/1688 [=============>................] - ETA: 3s - loss: 0.0098 - accuracy: 0.9980
811/1688 [=============>................] - ETA: 3s - loss: 0.0099 - accuracy: 0.9979
823/1688 [=============>................] - ETA: 3s - loss: 0.0099 - accuracy: 0.9979
835/1688 [=============>................] - ETA: 3s - loss: 0.0099 - accuracy: 0.9979
847/1688 [==============>...............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9979
859/1688 [==============>...............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9979
871/1688 [==============>...............] - ETA: 3s - loss: 0.0098 - accuracy: 0.9980
883/1688 [==============>...............] - ETA: 3s - loss: 0.0098 - accuracy: 0.9979
895/1688 [==============>...............] - ETA: 3s - loss: 0.0098 - accuracy: 0.9979
906/1688 [===============>..............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9980
918/1688 [===============>..............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9980
930/1688 [===============>..............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9980
942/1688 [===============>..............] - ETA: 3s - loss: 0.0098 - accuracy: 0.9979
954/1688 [===============>..............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9980
966/1688 [================>.............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9980
978/1688 [================>.............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9980
990/1688 [================>.............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9980
1002/1688 [================>.............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9980
1014/1688 [=================>............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9980
1026/1688 [=================>............] - ETA: 2s - loss: 0.0097 - accuracy: 0.9980
1038/1688 [=================>............] - ETA: 2s - loss: 0.0098 - accuracy: 0.9980
1050/1688 [=================>............] - ETA: 2s - loss: 0.0099 - accuracy: 0.9980
1061/1688 [=================>............] - ETA: 2s - loss: 0.0099 - accuracy: 0.9979
1073/1688 [==================>...........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9979
1085/1688 [==================>...........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9979
1097/1688 [==================>...........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9979
1109/1688 [==================>...........] - ETA: 2s - loss: 0.0101 - accuracy: 0.9978
1121/1688 [==================>...........] - ETA: 2s - loss: 0.0101 - accuracy: 0.9978
1133/1688 [===================>..........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9978
1145/1688 [===================>..........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9978
1157/1688 [===================>..........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9978
1169/1688 [===================>..........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9978
1181/1688 [===================>..........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9978
1193/1688 [====================>.........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9977
1205/1688 [====================>.........] - ETA: 2s - loss: 0.0104 - accuracy: 0.9977
1217/1688 [====================>.........] - ETA: 2s - loss: 0.0104 - accuracy: 0.9977
1229/1688 [====================>.........] - ETA: 2s - loss: 0.0104 - accuracy: 0.9977
1241/1688 [=====================>........] - ETA: 1s - loss: 0.0105 - accuracy: 0.9977
1253/1688 [=====================>........] - ETA: 1s - loss: 0.0106 - accuracy: 0.9977
1265/1688 [=====================>........] - ETA: 1s - loss: 0.0106 - accuracy: 0.9977
1277/1688 [=====================>........] - ETA: 1s - loss: 0.0105 - accuracy: 0.9977
1289/1688 [=====================>........] - ETA: 1s - loss: 0.0106 - accuracy: 0.9976
1301/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9976
1313/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9976
1325/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9976
1337/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9976
1349/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9976
1361/1688 [=======================>......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9976
1373/1688 [=======================>......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9976
1385/1688 [=======================>......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9975
1397/1688 [=======================>......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9975
1409/1688 [========================>.....] - ETA: 1s - loss: 0.0108 - accuracy: 0.9975
1421/1688 [========================>.....] - ETA: 1s - loss: 0.0108 - accuracy: 0.9975
1433/1688 [========================>.....] - ETA: 1s - loss: 0.0108 - accuracy: 0.9975
1445/1688 [========================>.....] - ETA: 1s - loss: 0.0107 - accuracy: 0.9975
1456/1688 [========================>.....] - ETA: 1s - loss: 0.0110 - accuracy: 0.9974
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1479/1688 [=========================>....] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1491/1688 [=========================>....] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0110 - accuracy: 0.9974
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1526/1688 [==========================>...] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1538/1688 [==========================>...] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0112 - accuracy: 0.9973
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0112 - accuracy: 0.9973
1573/1688 [==========================>...] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1643/1688 [============================>.] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1655/1688 [============================>.] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1667/1688 [============================>.] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1679/1688 [============================>.] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0111 - accuracy: 0.9974 - val_loss: 0.0503 - val_accuracy: 0.9865
Epoch 5/5
1/1688 [..............................] - ETA: 7s - loss: 0.0079 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0056 - accuracy: 0.9976
25/1688 [..............................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9987
37/1688 [..............................] - ETA: 7s - loss: 0.0056 - accuracy: 0.9992
49/1688 [..............................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9994
61/1688 [>.............................] - ETA: 7s - loss: 0.0050 - accuracy: 0.9995
73/1688 [>.............................] - ETA: 7s - loss: 0.0066 - accuracy: 0.9996
85/1688 [>.............................] - ETA: 7s - loss: 0.0071 - accuracy: 0.9993
97/1688 [>.............................] - ETA: 7s - loss: 0.0074 - accuracy: 0.9994
109/1688 [>.............................] - ETA: 7s - loss: 0.0072 - accuracy: 0.9994
120/1688 [=>............................] - ETA: 7s - loss: 0.0082 - accuracy: 0.9990
132/1688 [=>............................] - ETA: 6s - loss: 0.0092 - accuracy: 0.9986
144/1688 [=>............................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9980
156/1688 [=>............................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9982
168/1688 [=>............................] - ETA: 6s - loss: 0.0097 - accuracy: 0.9981
180/1688 [==>...........................] - ETA: 6s - loss: 0.0097 - accuracy: 0.9983
192/1688 [==>...........................] - ETA: 6s - loss: 0.0097 - accuracy: 0.9982
204/1688 [==>...........................] - ETA: 6s - loss: 0.0093 - accuracy: 0.9983
216/1688 [==>...........................] - ETA: 6s - loss: 0.0096 - accuracy: 0.9983
228/1688 [===>..........................] - ETA: 6s - loss: 0.0097 - accuracy: 0.9982
240/1688 [===>..........................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9980
252/1688 [===>..........................] - ETA: 6s - loss: 0.0100 - accuracy: 0.9980
264/1688 [===>..........................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9980
276/1688 [===>..........................] - ETA: 6s - loss: 0.0099 - accuracy: 0.9981
288/1688 [====>.........................] - ETA: 6s - loss: 0.0099 - accuracy: 0.9980
300/1688 [====>.........................] - ETA: 6s - loss: 0.0099 - accuracy: 0.9979
312/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9980
324/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9980
335/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9980
347/1688 [=====>........................] - ETA: 5s - loss: 0.0098 - accuracy: 0.9980
359/1688 [=====>........................] - ETA: 5s - loss: 0.0100 - accuracy: 0.9980
371/1688 [=====>........................] - ETA: 5s - loss: 0.0098 - accuracy: 0.9981
383/1688 [=====>........................] - ETA: 5s - loss: 0.0100 - accuracy: 0.9979
395/1688 [======>.......................] - ETA: 5s - loss: 0.0098 - accuracy: 0.9979
407/1688 [======>.......................] - ETA: 5s - loss: 0.0101 - accuracy: 0.9978
419/1688 [======>.......................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9978
431/1688 [======>.......................] - ETA: 5s - loss: 0.0101 - accuracy: 0.9978
443/1688 [======>.......................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9977
455/1688 [=======>......................] - ETA: 5s - loss: 0.0101 - accuracy: 0.9977
467/1688 [=======>......................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9977
478/1688 [=======>......................] - ETA: 5s - loss: 0.0106 - accuracy: 0.9975
490/1688 [=======>......................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9976
502/1688 [=======>......................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9976
514/1688 [========>.....................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9976
526/1688 [========>.....................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9977
538/1688 [========>.....................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9977
550/1688 [========>.....................] - ETA: 5s - loss: 0.0101 - accuracy: 0.9978
562/1688 [========>.....................] - ETA: 5s - loss: 0.0101 - accuracy: 0.9978
574/1688 [=========>....................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9977
586/1688 [=========>....................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9978
598/1688 [=========>....................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9978
610/1688 [=========>....................] - ETA: 4s - loss: 0.0104 - accuracy: 0.9976
622/1688 [==========>...................] - ETA: 4s - loss: 0.0103 - accuracy: 0.9976
634/1688 [==========>...................] - ETA: 4s - loss: 0.0103 - accuracy: 0.9976
646/1688 [==========>...................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
658/1688 [==========>...................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
670/1688 [==========>...................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9976
682/1688 [===========>..................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
694/1688 [===========>..................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9977
706/1688 [===========>..................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9977
718/1688 [===========>..................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9977
730/1688 [===========>..................] - ETA: 4s - loss: 0.0103 - accuracy: 0.9976
742/1688 [============>.................] - ETA: 4s - loss: 0.0103 - accuracy: 0.9976
754/1688 [============>.................] - ETA: 4s - loss: 0.0104 - accuracy: 0.9976
766/1688 [============>.................] - ETA: 4s - loss: 0.0105 - accuracy: 0.9974
778/1688 [============>.................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9973
789/1688 [=============>................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9973
801/1688 [=============>................] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
813/1688 [=============>................] - ETA: 3s - loss: 0.0105 - accuracy: 0.9973
825/1688 [=============>................] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
837/1688 [=============>................] - ETA: 3s - loss: 0.0105 - accuracy: 0.9973
849/1688 [==============>...............] - ETA: 3s - loss: 0.0105 - accuracy: 0.9973
861/1688 [==============>...............] - ETA: 3s - loss: 0.0105 - accuracy: 0.9974
873/1688 [==============>...............] - ETA: 3s - loss: 0.0104 - accuracy: 0.9974
885/1688 [==============>...............] - ETA: 3s - loss: 0.0105 - accuracy: 0.9975
896/1688 [==============>...............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
908/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
920/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9974
932/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
944/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
956/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
968/1688 [================>.............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9974
980/1688 [================>.............] - ETA: 3s - loss: 0.0105 - accuracy: 0.9974
992/1688 [================>.............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
1004/1688 [================>.............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
1016/1688 [=================>............] - ETA: 2s - loss: 0.0108 - accuracy: 0.9973
1028/1688 [=================>............] - ETA: 2s - loss: 0.0108 - accuracy: 0.9973
1040/1688 [=================>............] - ETA: 2s - loss: 0.0108 - accuracy: 0.9973
1052/1688 [=================>............] - ETA: 2s - loss: 0.0108 - accuracy: 0.9973
1064/1688 [=================>............] - ETA: 2s - loss: 0.0108 - accuracy: 0.9973
1076/1688 [==================>...........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9972
1088/1688 [==================>...........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1100/1688 [==================>...........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1112/1688 [==================>...........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9972
1124/1688 [==================>...........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1136/1688 [===================>..........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9972
1148/1688 [===================>..........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9972
1160/1688 [===================>..........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1172/1688 [===================>..........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9973
1184/1688 [====================>.........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9973
1196/1688 [====================>.........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9973
1208/1688 [====================>.........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9973
1220/1688 [====================>.........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9973
1232/1688 [====================>.........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9973
1244/1688 [=====================>........] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1256/1688 [=====================>........] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1268/1688 [=====================>........] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1280/1688 [=====================>........] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1292/1688 [=====================>........] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1304/1688 [======================>.......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1316/1688 [======================>.......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1328/1688 [======================>.......] - ETA: 1s - loss: 0.0109 - accuracy: 0.9972
1340/1688 [======================>.......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9972
1352/1688 [=======================>......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1364/1688 [=======================>......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1376/1688 [=======================>......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1388/1688 [=======================>......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1400/1688 [=======================>......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1412/1688 [========================>.....] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1423/1688 [========================>.....] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1435/1688 [========================>.....] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1447/1688 [========================>.....] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1459/1688 [========================>.....] - ETA: 1s - loss: 0.0108 - accuracy: 0.9973
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0109 - accuracy: 0.9972
1483/1688 [=========================>....] - ETA: 0s - loss: 0.0109 - accuracy: 0.9972
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0109 - accuracy: 0.9971
1506/1688 [=========================>....] - ETA: 0s - loss: 0.0109 - accuracy: 0.9972
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0109 - accuracy: 0.9972
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0109 - accuracy: 0.9971
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1554/1688 [==========================>...] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0109 - accuracy: 0.9971
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0109 - accuracy: 0.9972
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1638/1688 [============================>.] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1650/1688 [============================>.] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1662/1688 [============================>.] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1674/1688 [============================>.] - ETA: 0s - loss: 0.0110 - accuracy: 0.9971
1685/1688 [============================>.] - ETA: 0s - loss: 0.0109 - accuracy: 0.9971
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0109 - accuracy: 0.9971 - val_loss: 0.0495 - val_accuracy: 0.9870
<keras.callbacks.History object at 0x7fcaa35d9d60>
score = model_quantized.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after fine tuning:', score)
Test accuracy after fine tuning: 0.9861000180244446
3. Convert
3.1 Convert to Akida model
When the quantized model produces satisfactory performance, it can be converted to the native Akida format. The convert function returns a model in Akida format ready for inference.
As with Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized)
model_akida.summary()
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 5
______________________________________________
__________________________________________________________________
Layer (type) Output shape Kernel shape
=============== SW/conv2d-dequantizer_2 (Software) ===============
conv2d (InputConv2D) [13, 13, 32] (3, 3, 1, 32)
__________________________________________________________________
depthwise_conv2d (DepthwiseConv2D) [7, 7, 32] (3, 3, 32, 1)
__________________________________________________________________
conv2d_1 (Conv2D) [7, 7, 64] (1, 1, 32, 64)
__________________________________________________________________
dense (Dense2D) [1, 1, 10] (3136, 10)
__________________________________________________________________
dequantizer_2 (Dequantizer) [1, 1, 10] N/A
__________________________________________________________________
3.2. Check performance
accuracy = model_akida.evaluate(x_test, y_test)
print('Test accuracy after conversion:', accuracy)
# For non-regression purposes
assert accuracy > 0.96
Test accuracy after conversion: 0.98580002784729
3.3 Show predictions for a single image
Display one of the test images, such as the first image in the dataset from above, to visualize the output of the model.
# Test a single example
sample_image = 0
image = x_test[sample_image]
outputs = model_akida.predict(image.reshape(1, 28, 28, 1))
print('Input Label: %i' % y_test[sample_image])
f, axarr = plt.subplots(1, 2)
axarr[0].imshow(x_test[sample_image].reshape((28, 28)), cmap=cm.Greys_r)
axarr[0].set_title('Class %d' % y_test[sample_image])
axarr[1].bar(range(10), outputs.squeeze())
axarr[1].set_xticks(range(10))
plt.show()
print(outputs.squeeze())
Input Label: 7
[ -9.9294 -6.782947 -2.2589736 -1.0848838 -12.00255
-5.000862 -20.072525 6.6028986 -3.342186 0.10450193]
Consider the output from the model above. As is typical in backprop-trained models, the final layer is a Dense layer with one neuron for each of the 10 classes in the dataset. The goal of training is to maximize the response of the neuron corresponding to the label of each training sample while minimizing the responses of the other neurons.
In the bar chart above, you can see the outputs from all 10 neurons. It is easy to see that neuron 7 responds much more strongly than the others. The first sample is indeed a number 7.
4. GXNOR/MNIST
A more robust model called GXNOR/MNIST is provided in the model zoo It is inspired from the GXNOR-Net paper. It comes with its pretrained 2/2/1 helper for which the float training was done for 20 epochs, then the model was then gradually quantized following: 4/4/4 –> 4/4/2 –> 2/2/2 –> 2/2/1 with a 15 epochs QAT step at each quantization stage.
Total running time of the script: (1 minutes 34.628 seconds)