Suddenly, when restore checkpoint save by the TensorFlow R12 with old version, some errors occurs:

1
ValueError: Restore called with invalid save path

According to the TensorFlow v0.12.0 RC0’s release note:

New checkpoint format becomes the default in tf.train.Saver. Old V1 checkpoints continue to be readable; controlled by the write_version argument, tf.train.Saver now by default writes out in the new V2 format. It significantly reduces the peak memory required and latency incurred during restore.

The tf.train.Saver has a new checkpoint format with write_version flag V2, and old format with V1. For example:

old format (V1) new format (V2)
model.ckpt-12345 model.ckpt-12345.index
model.ckpt-12345.meta model.ckpt-12345.meta
model.ckpt-12345.data-00000-of-00001

What if we want to use the old trained checkpoint? So I look into the source file of saver.py, and find out the init function of the tf.train.saver is:

1
2
def __init__(self, write_version=saver_pb2.SaverDef.V2):
self._write_version = write_version

We can easily save trained checkpoint in the old way like this:

1
2
3
4
5
import tensorflow as tf
from tensorflow.core.protobuf import saver_pb2
...
saver = tf.train.Saver(write_version = saver_pb2.SaverDef.V1)
saver.save(sess, './model.ckpt', global_step = step)

However, we will get lots of WARNING … :

1
2
3
4
5
6
WARNING:tensorflow:*******************************************************
WARNING:tensorflow:TensorFlow's V1 checkpoint format has been deprecated.
WARNING:tensorflow:Consider switching to the more efficient V2 format:
WARNING:tensorflow: `tf.train.Saver(write_version=tf.train.SaverDef.V2)`
WARNING:tensorflow:now on by default.
WARNING:tensorflow:*******************************************************

So, once we migrate our tf code to R12, we’d better use the new checkpoint format for better performance.