R2Inference - TensorFlow

TensorFlow™ is an open-source software library for high-performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs). It is originally developed by researchers and engineers from the Google Brain team within Google’s AI organization. Tensorflow is widely used to develop machine learning and deep learning applications.

Installation

R2Inference TensorFlow backend depends on the C/C++ TensorFlow API. The installation process consists of downloading the library, extracting, and linking.

TensorFlow Python API and utilities can be installed with python-pip. These are not needed by R2Inference, but they are highly recommended if you need to generate models.

X86

You can install the C/C++ Tensorflow API for x86 using the Install TensorFlow for C guide from TensorFlow.

Please notice currently Gst-Inference only supports Tensorflow V1. The previous page show only the latest releases for TensorFlow, instructions will apply in the same way but you can download directly the V1 binaries from:

NVIDIA^®Jetson™ (TX1, TX2, Xavier, Nano)

For NVIDIA Jetson boards, download the prebuilt binaries for free from RidgeRun Online Store. Then run the following:

sudo tar -C /usr/ -xzf <libtensorflow>.tar.gz
sudo ldconfig

These TensorFlow binaries are built to use the CPU or the GPU.

i.MX 8

For i.MX8 boards, download the prebuilt binary for free from RidgeRun Online Store. Then run the following:

sudo tar -C /usr/ -xzf <libtensorflow>.tar.gz
sudo ldconfig

The TensorFlow binary is built to use the CPU. The GPU accelerated build is not available.

Generating a model for R2I

In Tensorflow, all file formats are based on protocol buffers. As a summary, protocol buffers (or protobuf, as referred on the documentation) are data structures for which there are a set of tools to generate classes in C, Python, and other languages in order to load, save, and access the data between the supported API's. More information about TensorFlow Model files can be found here. The steps to generate a graph model suitable for GstInference on the Tensorflow backend can be summarized in three main steps:

Save the graph structure that describes your model
Save checkpoint of your model training session (Session variables)
Combine the graph structure with the checkpoint data (this step is typically referred to as freezing the graph)

Saving a session with TensorFlow python API

In Tensorflow, you can use a saver object to handle the saving and restoring of your model graph metadata and the checkpoint (variables) data. In general terms, outside a Tensorflow session a graph contains only the information regarding the mathematical operation that is performed, while the variables are given a particular value inside a session. Typically, after training your model, you can use the saver object to save both your graph structure and data checkpoint. The following is an example when working on the Tensorflow default graph:

#! /usr/bin/env python3

import tensorflow as tf
import os

#file name is model_graph.py
dir = os.path.dirname(os.path.realpath(__file__))

default_saver = tf.train.Saver() 

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  
  
  # Perform your graph construction and training
  
  
  default_saver.save(sess, dir + '/data-all')

This will generate 4 files:

model_graph.chkp.meta: Graph data and metadata (operations, configurations, etc), allows to load a graph and retrain it.
model_graph.chkp.index: This file has a key-value table linking a tensor name and the location to find the corresponding data in the chkp.data files
model_graph.chkp.data-00000-of-00001: Holds all variables (which includes weights of the graph) from the session at different timestamps
checkpoint: A file that keeps a record of the latest checkpoint files saved

The most important files are the chkp.meta and chkp.data files. On a directory, with such files, you can use the freeze_graph.py method provided by Tensorflow resources on a directory with the files generated by the saver object in order to generate a protocol buffer file suitable for GstInference.

You can refer to R2Inference Model Zoo for pre-trained models suitable for evaluating GstInference.

Create a model using saved weights from a .ckpt file

In some resources, pre-trained file can be provided in a .ckpt format, this corresponds to and old version output of the of a saver object and is the equivalent of your .ckpt-data obtained by the current saver. In this example, a .ckpt file for the InceptionV1 graph is provided and it needs to be converted to a model suitable for GstInference. First you need to obtain the .ckpt file:

mkdir gstinference_tf_model
cd gstinference_tf_model
wget http://download.tensorflow.org/models/inception_v1_2016_08_28.tar.gz
tar -xzf inception_v1_2016_08_28.tar.gz

Then you need to associate the .ckpt file with the graph object. In case of the InceptionV1 graph, it can be obtained from within the Tensorflow resources:

#! /usr/bin/env python3
import numpy as np
import tensorflow as tf

from tensorflow.contrib.slim.nets import inception

slim = tf.contrib.slim

def unpack(name, image_size, num_classes):
  with tf.Graph().as_default():
    image = tf.placeholder("float", [1, image_size, image_size, 3], name="input")
    with slim.arg_scope(inception.inception_v1_arg_scope()):
        logits, _ = inception.inception_v1(image, num_classes, is_training=False, spatial_squeeze=False)
    probabilities = tf.nn.softmax(logits)
    init_fn = slim.assign_from_checkpoint_fn('inception_v1.ckpt', slim.get_model_variables('InceptionV1'))

    with tf.Session() as sess:
        init_fn(sess)
        saver = tf.train.Saver(tf.global_variables())
        saver.save(sess, "output/"+name)

unpack('inception-v1', 224, 1001)

Finally you can freeze the graph by either the freeze_graph.py or the following python3 script:

import tensorflow as tf
#Step 1
#import the model metagraph
saver = tf.train.import_meta_graph('./output/inception-v1.meta', clear_devices=True)
#make that as the default graph
graph = tf.get_default_graph()
input_graph_def = graph.as_graph_def()
sess = tf.Session()
#now restore the variables
saver.restore(sess, "./inception_v1.ckpt")

#Step 2
# Find the output name
graph = tf.get_default_graph()
for op in graph.get_operations():
      print (op.name)

      #Step 3
      from tensorflow.python.platform import gfile
      from tensorflow.python.framework import graph_util

      output_node_names="InceptionV1/Logits/Predictions/Reshape_1"
      output_graph_def = graph_util.convert_variables_to_constants(
        sess, # The session
        input_graph_def, # input_graph_def is useful for retrieving the nodes
        output_node_names.split(",")  )

#Step 4
#output folder
output_fld ='./'
#output pb file name
output_model_file = 'inceptionv1_tensorflow.pb'
from tensorflow.python.framework import graph_io
#write the graph
graph_io.write_graph(output_graph_def, output_fld, output_model_file, as_text=False)

Tools

Tensorboard

TensorBoard is a visualization tool for TensorFlow. You can use TensorBoard to visualize your TensorFlow graph, plot quantitative metrics about the execution of your graph, and show additional data like images that pass through it. To use TensorBoard you simply need to install TensorFlow core. Installing TensorFlow via pip should also automatically install TensorBoard. This tool is especially useful to determine the input and output layer name of undocumented graphs. TensorBoard can load any TensorFlow checkpoint generated with the same version (loading a checkpoint generated with a different Tensorflow version will result in errors).

tensorboard --logdir=route/to/checkpoint/dir

You will get a message similar to this:

TensorBoard 1.10.0 at http://mtaylor-laptop:6006 (Press CTRL+C to quit)

Open that address in your browser, go to graph and analyze the graph to determine the output node name. In this example the output node name is ArgMax because it's input is the resnet_model/final_dense signal.

API

You can find the full documentation of the C API and Python API at

R2Inference uses only the C API and R2Inference takes care of the session, loading the graph, and executing. Because of this, we will only take a look at the options that you can change when using the C API through R2Inference.

R2Inference changes the options of the framework via the "IParameters" class. First you need to create an object:

r2i::RuntimeError error;
std::shared_ptr<r2i::IParameters> parameters = factory->MakeParameters (error);

Then call the "Set" or "Get" virtual functions:

parameters->Set(<option>, <value>)
parameters->Get(<option>, <value>)

Tensorflow Options

Property	C API Counterpart	Value	Operation	Description
version	TF_Version()	String	Read only	Tensorflow API version
output-layer	TF_GraphOperationByName	String	R/W	Name of the output layer of the graph
input-layer	TF_GraphOperationByName	String	R/W	Name of the input layer of the graph.

Previous: Supported_backends

Index

Next: Supported_backends/TensorFlow-Lite

❯