Skip to content
Snippets Groups Projects
Commit c2e995b3 authored by Simone Rossi's avatar Simone Rossi
Browse files

[add] Add lab2 files 


Signed-off-by: Simone Rossi's avatarSimone Rossi <simone.rossi@eurecom.fr>
parent bd415823
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
DL_lab2/MNIST_data/t10k-images-idx3-ubyte.gz 0 → 100644
+ 0
− 0
View file @ c2e995b3
File added
DL_lab2/MNIST_data/t10k-labels-idx1-ubyte.gz 0 → 100644
+ 0
− 0
View file @ c2e995b3
File added
DL_lab2/MNIST_data/train-images-idx3-ubyte.gz 0 → 100644
+ 0
− 0
View file @ c2e995b3
File added
DL_lab2/MNIST_data/train-labels-idx1-ubyte.gz 0 → 100644
+ 0
− 0
View file @ c2e995b3
File added
DL_lab2/lab2_students.ipynb 0 → 100644
+ 488
− 0
View file @ c2e995b3
%% Cell type:markdown id: tags:
<h1 style="text-align:center">Deep Learning </h1>
<h1 style="text-align:center"> Lab Session 2 - 1.5 Hours </h1>
<h1 style="text-align:center"> Convolutional Neural Network (CNN) for Handwritten Digits Recognition</h1>
%% Cell type:markdown id: tags:
The aim of this session is to practice with Convolutional Neural Networks. Each group should fill and run appropriate notebook cells.
Follow instructions step by step until the end and submit your complete notebook as an archive (tar -cf groupXnotebook.tar DL_lab2/).
Do not forget to run all your cells before generating your final report and do not forget to include the names of all participants in the group. The lab session should be completed by May 29th 2019 (23:59:59 CET).
%% Cell type:markdown id: tags:
# Introduction
%% Cell type:markdown id: tags:
In the last Lab Session, you built a Multilayer Perceptron for recognizing hand-written digits from the MNIST data-set. The best achieved accuracy on testing data was about 97%. Can you do better than these results using a deep CNN ?
In this Lab Session, you will build, train and optimize in TensorFlow one of the early Convolutional Neural Networks, **LeNet-5**, to go to more than 99% of accuracy.
%% Cell type:markdown id: tags:
# Load MNIST Data in TensorFlow
Run the cell below to load the MNIST data that comes with TensorFlow. You will use this data in **Section 1** and **Section 2**.
%% Cell type:code id: tags:
``` python
import tensorflow as tf
import numpy as np
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
X_train, y_train = mnist.train.images, mnist.train.labels
X_validation, y_validation = mnist.validation.images, mnist.validation.labels
X_test, y_test = mnist.test.images, mnist.test.labels
print("Image Shape: {}".format(X_train[0].shape))
print("Training Set: {} samples".format(len(X_train)))
print("Validation Set: {} samples".format(len(X_validation)))
print("Test Set: {} samples".format(len(X_test)))
epsilon = 1e-10 # this is a parameter you will use later
```
%% Cell type:markdown id: tags:
# Section 1 : My First Model in TensorFlow
%% Cell type:markdown id: tags:
Before starting with CNN, let's train and test in TensorFlow the example
**y=softmax(Wx+b)** seen in the first lab.
This model reaches an accuracy of about 92 %.
You will also learn how to launch the TensorBoard https://www.tensorflow.org/get_started/summaries_and_tensorboard to visualize the computation graph, statistics and learning curves.
%% Cell type:markdown id: tags:
<b> Part 1 </b> : Read carefully the code in the cell below. Run it to perform training.
%% Cell type:code id: tags:
``` python
#STEP 1
# Parameters
learning_rate = 0.01
training_epochs = 40
batch_size = 128
display_step = 1
logs_path = 'log_files/' # useful for tensorboard
# tf Graph Input: mnist data image of shape 28*28=784
x = tf.placeholder(tf.float32, [None, 784], name='InputData')
# 0-9 digits recognition, 10 classes
y = tf.placeholder(tf.float32, [None, 10], name='LabelData')
# Set model weights
W = tf.Variable(tf.zeros([784, 10]), name='Weights')
b = tf.Variable(tf.zeros([10]), name='Bias')
# Construct model and encapsulating all ops into scopes, making Tensorboard's Graph visualization more convenient
with tf.name_scope('Model'):
# Model
pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax
with tf.name_scope('Loss'):
# Minimize error using cross entropy
# We use tf.clip_by_value to avoid having too low numbers in the log function
cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(tf.clip_by_value(pred, epsilon, 1.0)), reduction_indices=1))
with tf.name_scope('SGD'):
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
with tf.name_scope('Accuracy'):
# Accuracy
acc = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
acc = tf.reduce_mean(tf.cast(acc, tf.float32))
# Initializing the variables
init = tf.global_variables_initializer()
# Create a summary to monitor cost tensor
tf.summary.scalar("Loss", cost)
# Create a summary to monitor accuracy tensor
tf.summary.scalar("Accuracy", acc)
# Merge all summaries into a single op
merged_summary_op = tf.summary.merge_all()
#STEP 2
# Launch the graph for training
with tf.Session() as sess:
sess.run(init)
# op to write logs to Tensorboard
summary_writer = tf.summary.FileWriter(logs_path, graph=tf.get_default_graph())
# Training cycle
for epoch in range(training_epochs):
avg_cost = 0.
total_batch = int(mnist.train.num_examples/batch_size)
# Loop over all batches
for i in range(total_batch):
batch_xs, batch_ys = mnist.train.next_batch(batch_size, shuffle=(i==0))
# Run optimization op (backprop), cost op (to get loss value)
# and summary nodes
_, c, summary = sess.run([optimizer, cost, merged_summary_op],
feed_dict={x: batch_xs, y: batch_ys})
# Write logs at every iteration
summary_writer.add_summary(summary, epoch * total_batch + i)
# Compute average loss
avg_cost += c / total_batch
# Display logs per epoch step
if (epoch+1) % display_step == 0:
print("Epoch: ", '%02d' % (epoch+1), " =====> Loss=", "{:.9f}".format(avg_cost))
print("Optimization Finished!")
summary_writer.flush()
# Test model
# Calculate accuracy
print("Accuracy:", acc.eval({x: mnist.test.images, y: mnist.test.labels}))
```
%% Cell type:markdown id: tags:
<b> Part 2 </b>: Using Tensorboard, we can now visualize the created graph, giving you an overview of your architecture and how all of the major components are connected. You can also see and analyse the learning curves.
To launch tensorBoard:
- Open a Terminal and run the command line **"tensorboard --logdir=lab_2/log_files/"**
- Click on "Tensorboard web interface" in Zoe
Enjoy It !!
%% Cell type:markdown id: tags:
# Section 2 : The 99% MNIST Challenge !
%% Cell type:markdown id: tags:
<b> Part 1 </b> : LeNet5 implementation
%% Cell type:markdown id: tags:
You are now familar with **TensorFlow** and **TensorBoard**. In this section, you are to build, train and test the baseline [LeNet-5](http://yann.lecun.com/exdb/lenet/) model for the MNIST digits recognition problem.
Then, you will make some optimizations to get more than 99% of accuracy.
For more informations, have a look at this list of results: http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html
%% Cell type:markdown id: tags:
<img src="lenet.png" width="800" height="600" align="center">
The LeNet architecture takes a 28x28xC image as input, where C is the number of color channels. Since MNIST images are grayscale, C is 1 in this case.
--------------------------
**Layer 1 - Convolution (5x5):** The output shape should be 28x28x6. **Activation:** ReLU. **MaxPooling:** The output shape should be 14x14x6.
**Layer 2 - Convolution (5x5):** The output shape should be 10x10x16. **Activation:** ReLU. **MaxPooling:** The output shape should be 5x5x16.
**Flatten:** Flatten the output shape of the final pooling layer such that it's 1D instead of 3D. You may need to use tf.reshape.
**Layer 3 - Fully Connected:** This should have 120 outputs. **Activation:** ReLU.
**Layer 4 - Fully Connected:** This should have 84 outputs. **Activation:** ReLU.
**Layer 5 - Fully Connected:** This should have 10 outputs. **Activation:** softmax.
%% Cell type:markdown id: tags:
<b> Question 2.1.1 </b> Implement the Neural Network architecture described above.
For that, your will use classes and functions from https://www.tensorflow.org/api_docs/python/tf/nn.
We give you some helper functions for weigths and bias initilization. Also you can refer to section 1.
%% Cell type:code id: tags:
``` python
# Functions for weigths and bias initilization
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variable(shape):
initial = tf.constant(0., shape=shape)
return tf.Variable(initial)
```
%% Cell type:code id: tags:
``` python
def LeNet5_Model(image):
# your inmplementation goes here
pass
```
%% Cell type:markdown id: tags:
<b> Question 2.1.2. </b> Calculate the number of parameters of this model
%% Cell type:markdown id: tags:
Your answer goes here in details
%% Cell type:markdown id: tags:
<b> Question 2.1.3. </b> Define your model, its accuracy and the loss function according to the following parameters (you can look at Section 1 to see what is expected):
Learning rate: 0.001
Loss Fucntion: Cross-entropy
Optimizer: tf.train.GradientDescentOptimizer
Number of epochs: 40
Batch size: 128
%% Cell type:code id: tags:
``` python
tf.reset_default_graph() # reset the default graph before defining a new model
# Parameters
learning_rate =
training_epochs =
batch_size =
logs_path = 'log_files/'
# Model, loss function and accuracy
```
%% Cell type:markdown id: tags:
<b> Question 2.1.4. </b> Implement the evaluation function for accuracy computation
%% Cell type:code id: tags:
``` python
def evaluate(logits, labels):
# logits will be the outputs of your model, labels will be one-hot vectors corresponding to the actual labels
# logits and labels are numpy arrays
# this function should return the accuracy of your model
pass
```
%% Cell type:markdown id: tags:
<b> Question 2.1.5. </b> Implement training pipeline and run the training data through it to train the model.
- Before each epoch, shuffle the training set.
- Print the loss per mini batch and the training/validation accuracy per epoch. (Display results every 100 epochs)
- Save the model after training
- Print after training the final testing accuracy
%% Cell type:code id: tags:
``` python
# Initializing the variables
init = tf.global_variables_initializer()
# Create a summary to monitor cost tensor
tf.summary.scalar("Loss_LeNet-5_SGD", cost)
# Create a summary to monitor accuracy tensor
tf.summary.scalar("Accuracy_LeNet-5_SGD", acc)
# Merge all summaries into a single op
merged_summary_op = tf.summary.merge_all()
# Initializing the variables
def train(init, sess, logs_path, n_epochs, batch_size, optimizer, cost, merged_summary_op):
# optimizer and cost are the same kinds of objects as in Section 1
# Train your model
pass
# Print the accuracy on testing data
pass
with tf.Session() as sess:
train(init, sess, logs_path, training_epochs, batch_size, optimizer, cost, merged_summary_op)
```
%% Cell type:markdown id: tags:
<b> Question 2.1.6 </b> : Use TensorBoard to visualise and save loss and accuracy curves.
You will save figures in the folder **"lab_2/MNIST_figures"** and display them in your notebook.
%% Cell type:markdown id: tags:
Please put your loss and accuracy curves here.
%% Cell type:markdown id: tags:
<b> Part 2 </b> : LeNET 5 Optimization
%% Cell type:markdown id: tags:
<b> Question 2.2.1 </b>
- Retrain your network with AdamOptimizer and then fill the table above:
| Optimizer | Gradient Descent | AdamOptimizer |
|----------------------|--------------------|---------------------|
| Testing Accuracy | ??? | ??? |
| Training Time | ??? | ??? |
- Which optimizer gives the best accuracy on test data?
**Your answer:** ...
%% Cell type:code id: tags:
``` python
tf.reset_default_graph()
# your implementation goes here
```
%% Cell type:markdown id: tags:
<b> Question 2.2.2</b> Try to add dropout (keep_prob = 0.75) before the first fully connected layer. You will use tf.nn.dropout for that purpose. What accuracy do you achieve on testing data?
**Accuracy achieved on testing data:** ...
%% Cell type:code id: tags:
``` python
def LeNet5_Model_Dropout(image):
# your implementation goes here
tf.reset_default_graph()
# your implementation goes here
```
DL_lab2/lenet.png 0 → 100644
+ 0
− 0
View file @ c2e995b3
DL_lab2/lenet.png

65 KiB

0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment