Recording Validation Loss While Training

#1

Hi all, happy to join the community. I’ve been working on a project and bashing my head for a week trying to solve this.

I’m training a model that is built on Keras and while training, trying to track various metrics. I’ve been able to use the deepchem library to successfully track what is inside the code block below, however for the life of me I cannot figure out how to record the training/validation loss at multiple points during training.

I’ve seen other posts from a few years back that seem to ask the same questions.
Some background:

  • I’m using Google Colab Pro
  • I’ve looked into various Keras tools like ‘history’, ‘CSV logger’, and ModelCheckpoint and cannot seem to get anything to work.

Any thoughts / suggestions would be greatly appreciated!

m1 = dc.metrics.Metric(dc.metrics.roc_auc_score)
m2 = dc.metrics.Metric(dc.metrics.accuracy_score)
m3 = dc.metrics.Metric(dc.metrics.balanced_accuracy_score)

f = open('all_results.txt', 'a')

model = dc.models.GraphConvModel(n_tasks=1, mode='classification', 
                                    graph_conv_layers = graph_conv_layers,
                                    batchnorm = batchnorm, dropout = dropout,
                                    dense_layer_size = dense_layer_size,
                                    batchsize = batch_size)

callback = dc.models.ValidationCallback(valid, 1000, [m1, m2, m3], save_dir='.', output_file = f)
model.fit(train, nb_epoch=epochs, callbacks = callback) 

f.close()
1 Like
#2

I believe there is a way to do this with ValidationCallback but I often just set up a simple loop in my code. Something like

for i in range(n_epochs):
  model.fit(train, nb_epoch=1)
  metrics = model.evaluate(validation,...)

It’s a bit kludgy but very robust and simple

2 Likes
#3

This is great thank you so much! If others are interested in what I did, I’ve added the code snippet below. I used the sklearn ‘log_loss’ function to easily compare my validation predictions to the true labels, then added this value to my metrics dictionary.

all_metrics = {}

for i in range(epochs):
    model.fit(train, nb_epoch=1) 

    # every 5 epochs evaluate the model
    if i % 5 == 0:
        validation_preds = model.predict(valid)
        metrics = model.evaluate(valid, metrics=[m1, m2, m3])
        metrics['log_loss'] = log_loss(valid.y, [i[0] for i in validation_preds])
        all_metrics[i+1] = metrics
1 Like