Keras checkpoints delete after training

Hi all,

I’m having some difficulty saving weights during training of my GraphConvModel and would love any advice. I’ve tested out a few different scenarios.

  1. Running my models in a .ipynb notebook, fitting the model, and saving the checkpoints works as expected. The checkpoints are saved in the specified model directory.
  2. Running my models in .py files also save the checkpoints, however as soon as the python script is completed the checkpoints disappear.

I’ve tried messing around with parameters including the number of checkpoints to keep (passing in nothing or a random number), however I am unable to keep the checkpoints throughout training with my python scripts. Any thoughts?

An example of my outputs during training are in the screenshot. Immediately following execution the directory disappears.

The subset of code involved in the model training is included below.

`model = dc.models.GraphConvModel(n_tasks=1, mode='classification',

    all_metrics = {}
    for i in range(1, 11):, nb_epoch=1)

        # every n epochs evaluate the model
        if i % 2 == 0:
            validation_preds = model.predict(valid)
            metrics = model.evaluate(valid, metrics=dc_metrics())
            metrics['log_loss'] = log_loss(valid.y, [i[0] for i in validation_preds])

            all_metrics = model_evaluation.prediction_metrics(metrics, validation_preds,
                                             valid.y, probability_threshold)

            print(f'\nEpoch {i}\n------------------------')
            ## every 10 epochs save the weights

Screenshot 2023-12-19 at 8.31.43 PM

Thanks in advance!

Some suggestions which might work:

  • Try specifying model_dir along with model parameters - this will specify a directory to save the checkpoints to keep
  • Try setting max_checkpoints_to_keep to set the number of checkpoints to store