Content-Based Filtering Using Neural Networks

This notebook relies on files created in the content_based_preproc.ipynb notebook. Be sure to run the code in there before completing this notebook.
Also, we'll be using the python3 kernel from here on out so don't forget to change the kernel if it's still Python2.

This lab illustrates:

  1. how to build feature columns for a model using tf.feature_column
  2. how to create custom evaluation metrics and add them to Tensorboard
  3. how to train a model and make predictions with the saved model

Tensorflow Hub should already be installed. You can check that it is by using "pip freeze".

In [1]:
%%bash
pip freeze | grep tensor
tensorboard==1.13.1
tensorflow==1.13.1
tensorflow-data-validation==0.21.5
tensorflow-datasets==1.2.0
tensorflow-estimator==1.13.0
tensorflow-hub==0.4.0
tensorflow-io==0.8.1
tensorflow-metadata==0.21.2
tensorflow-model-analysis==0.21.6
tensorflow-probability==0.8.0
tensorflow-serving-api==1.15.0
tensorflow-transform==0.21.2

Let's make sure we install the necessary version of tensorflow-hub. After doing the pip install below, click "Reset Session" on the notebook so that the Python environment picks up the new packages.

In [2]:
!pip3 install tensorflow-hub==0.4.0
!pip3 install --upgrade tensorflow==1.13.1
Requirement already satisfied: tensorflow-hub==0.4.0 in /opt/conda/lib/python3.7/site-packages (0.4.0)
Requirement already satisfied: protobuf>=3.4.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-hub==0.4.0) (3.11.4)
Requirement already satisfied: six>=1.10.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-hub==0.4.0) (1.14.0)
Requirement already satisfied: numpy>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-hub==0.4.0) (1.18.3)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.4.0->tensorflow-hub==0.4.0) (46.1.3.post20200325)
Requirement already up-to-date: tensorflow==1.13.1 in /opt/conda/lib/python3.7/site-packages (1.13.1)
Requirement already satisfied, skipping upgrade: tensorboard<1.14.0,>=1.13.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (1.13.1)
Requirement already satisfied, skipping upgrade: absl-py>=0.1.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (0.8.1)
Requirement already satisfied, skipping upgrade: keras-preprocessing>=1.0.5 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (1.1.0)
Requirement already satisfied, skipping upgrade: protobuf>=3.6.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (3.11.4)
Requirement already satisfied, skipping upgrade: gast>=0.2.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (0.2.2)
Requirement already satisfied, skipping upgrade: grpcio>=1.8.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (1.28.1)
Requirement already satisfied, skipping upgrade: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (1.1.0)
Requirement already satisfied, skipping upgrade: keras-applications>=1.0.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (1.0.8)
Requirement already satisfied, skipping upgrade: astor>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (0.8.1)
Requirement already satisfied, skipping upgrade: tensorflow-estimator<1.14.0rc0,>=1.13.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (1.13.0)
Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (1.18.3)
Requirement already satisfied, skipping upgrade: wheel>=0.26 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (0.34.2)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==1.13.1) (1.14.0)
Requirement already satisfied, skipping upgrade: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow==1.13.1) (3.2.1)
Requirement already satisfied, skipping upgrade: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow==1.13.1) (1.0.1)
Requirement already satisfied, skipping upgrade: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.1->tensorflow==1.13.1) (46.1.3.post20200325)
Requirement already satisfied, skipping upgrade: h5py in /opt/conda/lib/python3.7/site-packages (from keras-applications>=1.0.6->tensorflow==1.13.1) (2.10.0)
Requirement already satisfied, skipping upgrade: mock>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-estimator<1.14.0rc0,>=1.13.0->tensorflow==1.13.1) (3.0.5)
In [3]:
import os
import tensorflow as tf
import numpy as np
import tensorflow_hub as hub
import shutil

PROJECT = 'my-project' # REPLACE WITH YOUR PROJECT ID
BUCKET = 'my-bucket' # REPLACE WITH YOUR BUCKET NAME
REGION = 'us-west1' # REPLACE WITH YOUR BUCKET REGION e.g. us-central1

# do not change these
os.environ['PROJECT'] = PROJECT
os.environ['BUCKET'] = BUCKET
os.environ['REGION'] = REGION
os.environ['TFVERSION'] = '1.13.1'
In [4]:
%%bash
gcloud config set project $PROJECT
gcloud config set compute/region $REGION
Updated property [core/project].
Updated property [compute/region].

Build the feature columns for the model.

To start, we'll load the list of categories, authors and article ids we created in the previous Create Datasets notebook.

In [5]:
categories_list = open("categories.txt").read().splitlines()
authors_list = open("authors.txt").read().splitlines()
content_ids_list = open("content_ids.txt").read().splitlines()
mean_months_since_epoch = 523

In the cell below we'll define the feature columns to use in our model. If necessary, remind yourself the various feature columns to use.
For the embedded_title_column feature column, use a Tensorflow Hub Module to create an embedding of the article title. Since the articles and titles are in German, you'll want to use a German language embedding module.
Explore the text embedding Tensorflow Hub modules available here. Filter by setting the language to 'German'. The 50 dimensional embedding should be sufficient for our purposes.

In [6]:
embedded_title_column = hub.text_embedding_column(
    key="title", 
    module_spec="https://tfhub.dev/google/nnlm-de-dim50/1",
    trainable=False)

content_id_column = tf.feature_column.categorical_column_with_hash_bucket(
    key="content_id",
    hash_bucket_size= len(content_ids_list) + 1)
embedded_content_column = tf.feature_column.embedding_column(
    categorical_column=content_id_column,
    dimension=10)

author_column = tf.feature_column.categorical_column_with_hash_bucket(key="author",
    hash_bucket_size=len(authors_list) + 1)
embedded_author_column = tf.feature_column.embedding_column(
    categorical_column=author_column,
    dimension=3)

category_column_categorical = tf.feature_column.categorical_column_with_vocabulary_list(
    key="category",
    vocabulary_list=categories_list,
    num_oov_buckets=1)
category_column = tf.feature_column.indicator_column(category_column_categorical)

months_since_epoch_boundaries = list(range(400,700,20))
months_since_epoch_column = tf.feature_column.numeric_column(
    key="months_since_epoch")
months_since_epoch_bucketized = tf.feature_column.bucketized_column(
    source_column = months_since_epoch_column,
    boundaries = months_since_epoch_boundaries)

crossed_months_since_category_column = tf.feature_column.indicator_column(tf.feature_column.crossed_column(
  keys = [category_column_categorical, months_since_epoch_bucketized], 
  hash_bucket_size = len(months_since_epoch_boundaries) * (len(categories_list) + 1)))

feature_columns = [embedded_content_column,
                   embedded_author_column,
                   category_column,
                   embedded_title_column,
                   crossed_months_since_category_column] 

Create the input function.

Next we'll create the input function for our model. This input function reads the data from the csv files we created in the previous labs.

In [7]:
record_defaults = [["Unknown"], ["Unknown"],["Unknown"],["Unknown"],["Unknown"],[mean_months_since_epoch],["Unknown"]]
column_keys = ["visitor_id", "content_id", "category", "title", "author", "months_since_epoch", "next_content_id"]
label_key = "next_content_id"
def read_dataset(filename, mode, batch_size = 512):
  def _input_fn():
      def decode_csv(value_column):
          columns = tf.decode_csv(value_column,record_defaults=record_defaults)
          features = dict(zip(column_keys, columns))          
          label = features.pop(label_key)         
          return features, label

      # Create list of files that match pattern
      file_list = tf.gfile.Glob(filename)

      # Create dataset from file list
      dataset = tf.data.TextLineDataset(file_list).map(decode_csv)

      if mode == tf.estimator.ModeKeys.TRAIN:
          num_epochs = None # indefinitely
          dataset = dataset.shuffle(buffer_size = 10 * batch_size)
      else:
          num_epochs = 1 # end-of-input after this

      dataset = dataset.repeat(num_epochs).batch(batch_size)
      return dataset.make_one_shot_iterator().get_next()
  return _input_fn

Create the model and train/evaluate

Next, we'll build our model which recommends an article for a visitor to the Kurier.at website. Look through the code below. We use the input_layer feature column to create the dense input layer to our network. This is just a sigle layer network where we can adjust the number of hidden units as a parameter.

Currently, we compute the accuracy between our predicted 'next article' and the actual 'next article' read next by the visitor. We'll also add an additional performance metric of top 10 accuracy to assess our model. To accomplish this, we compute the top 10 accuracy metric, add it to the metrics dictionary below and add it to the tf.summary so that this value is reported to Tensorboard as well.

In [8]:
def model_fn(features, labels, mode, params):
  net = tf.feature_column.input_layer(features, params['feature_columns'])
  for units in params['hidden_units']:
        net = tf.layers.dense(net, units=units, activation=tf.nn.relu)
   # Compute logits (1 per class).
  logits = tf.layers.dense(net, params['n_classes'], activation=None) 

  predicted_classes = tf.argmax(logits, 1)
  from tensorflow.python.lib.io import file_io
    
  with file_io.FileIO('content_ids.txt', mode='r') as ifp:
    content = tf.constant([x.rstrip() for x in ifp])
  predicted_class_names = tf.gather(content, predicted_classes)
  if mode == tf.estimator.ModeKeys.PREDICT:
    predictions = {
        'class_ids': predicted_classes[:, tf.newaxis],
        'class_names' : predicted_class_names[:, tf.newaxis],
        'probabilities': tf.nn.softmax(logits),
        'logits': logits,
    }
    return tf.estimator.EstimatorSpec(mode, predictions=predictions)
  table = tf.contrib.lookup.index_table_from_file(vocabulary_file="content_ids.txt")
  labels = table.lookup(labels)
  # Compute loss.
  loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)

  # Compute evaluation metrics.
  accuracy = tf.metrics.accuracy(labels=labels,
                                 predictions=predicted_classes,
                                 name='acc_op')
  top_10_accuracy = tf.metrics.mean(tf.nn.in_top_k(predictions=logits, 
                                                   targets=labels, 
                                                   k=10))
  
  metrics = {
    'accuracy': accuracy,
    'top_10_accuracy' : top_10_accuracy}
  
  tf.summary.scalar('accuracy', accuracy[1])
  tf.summary.scalar('top_10_accuracy', top_10_accuracy[1])

  if mode == tf.estimator.ModeKeys.EVAL:
      return tf.estimator.EstimatorSpec(
          mode, loss=loss, eval_metric_ops=metrics)

  # Create training op.
  assert mode == tf.estimator.ModeKeys.TRAIN

  optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)
  train_op = optimizer.minimize(loss, global_step=tf.train.get_global_step())
  return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op)

Train and Evaluate

In [9]:
outdir = 'content_based_model_trained'
shutil.rmtree(outdir, ignore_errors = True) # start fresh each time
tf.summary.FileWriterCache.clear() # ensure filewriter cache is clear for TensorBoard events file
estimator = tf.estimator.Estimator(
    model_fn=model_fn,
    model_dir = outdir,
    params={
     'feature_columns': feature_columns,
      'hidden_units': [200, 100, 50],
      'n_classes': len(content_ids_list)
    })

train_spec = tf.estimator.TrainSpec(
    input_fn = read_dataset("training_set.csv", tf.estimator.ModeKeys.TRAIN),
    max_steps = 2000)

eval_spec = tf.estimator.EvalSpec(
    input_fn = read_dataset("test_set.csv", tf.estimator.ModeKeys.EVAL),
    steps = None,
    start_delay_secs = 30,
    throttle_secs = 60)

tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f82c82af710>, '_model_dir': 'content_based_model_trained', '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_session_creation_timeout_secs': 7200, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_experimental_max_worker_delay_secs': None, '_evaluation_master': '', '_eval_distribute': None, '_train_distribute': None, '_master': ''}
INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f82c82af710>, '_model_dir': 'content_based_model_trained', '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_session_creation_timeout_secs': 7200, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_experimental_max_worker_delay_secs': None, '_evaluation_master': '', '_eval_distribute': None, '_train_distribute': None, '_master': ''}
INFO:tensorflow:Not using Distribute Coordinator.
INFO:tensorflow:Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/training/training_util.py:236: initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/training/training_util.py:236: initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_estimator/python/estimator/api/_v1/estimator/__init__.py:12: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.

WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_estimator/python/estimator/api/_v1/estimator/__init__.py:12: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.

WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.decode_csv is deprecated. Please use tf.io.decode_csv instead.

WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.decode_csv is deprecated. Please use tf.io.decode_csv instead.

WARNING:tensorflow:From <ipython-input-7-fe0036610997>:25: make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
WARNING:tensorflow:From <ipython-input-7-fe0036610997>:25: make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:206: _get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:206: _get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:3182: _get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:3182: _get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:3122: _num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:3122: _num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/ops/embedding_ops.py:802: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/ops/embedding_ops.py:802: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:207: _variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:207: _variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:206: _get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:206: _get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4300: _get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4300: _get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4118: _get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4118: _get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4118: _get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4118: _get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column.py:2158: _transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4271: _variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4271: _variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4326: _num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4326: _num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4326: _num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4326: _num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From <ipython-input-8-6b888c179a6d>:4: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
WARNING:tensorflow:From <ipython-input-8-6b888c179a6d>:4: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/layers/core.py:187: apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/layers/core.py:187: apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/training/adagrad.py:76: calling __init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /opt/conda/envs/py27/lib/python2.7/site-packages/tensorflow_core/python/training/adagrad.py:76: calling __init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into content_based_model_trained/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into content_based_model_trained/model.ckpt.
INFO:tensorflow:loss = 9.657072, step = 1
INFO:tensorflow:loss = 9.657072, step = 1
INFO:tensorflow:global_step/sec: 4.56369
INFO:tensorflow:global_step/sec: 4.56369
INFO:tensorflow:loss = 5.3729877, step = 101 (21.922 sec)
INFO:tensorflow:loss = 5.3729877, step = 101 (21.922 sec)
INFO:tensorflow:global_step/sec: 4.7928
INFO:tensorflow:global_step/sec: 4.7928
INFO:tensorflow:loss = 5.245719, step = 201 (20.862 sec)
INFO:tensorflow:loss = 5.245719, step = 201 (20.862 sec)
INFO:tensorflow:global_step/sec: 4.78348
INFO:tensorflow:global_step/sec: 4.78348
INFO:tensorflow:loss = 5.148116, step = 301 (20.906 sec)
INFO:tensorflow:loss = 5.148116, step = 301 (20.906 sec)
INFO:tensorflow:global_step/sec: 4.74539
INFO:tensorflow:global_step/sec: 4.74539
INFO:tensorflow:loss = 5.091365, step = 401 (21.069 sec)
INFO:tensorflow:loss = 5.091365, step = 401 (21.069 sec)
INFO:tensorflow:global_step/sec: 4.81149
INFO:tensorflow:global_step/sec: 4.81149
INFO:tensorflow:loss = 5.107406, step = 501 (20.784 sec)
INFO:tensorflow:loss = 5.107406, step = 501 (20.784 sec)
INFO:tensorflow:global_step/sec: 4.77907
INFO:tensorflow:global_step/sec: 4.77907
INFO:tensorflow:loss = 4.9904337, step = 601 (20.931 sec)
INFO:tensorflow:loss = 4.9904337, step = 601 (20.931 sec)
INFO:tensorflow:global_step/sec: 4.73306
INFO:tensorflow:global_step/sec: 4.73306
INFO:tensorflow:loss = 5.130045, step = 701 (21.128 sec)
INFO:tensorflow:loss = 5.130045, step = 701 (21.128 sec)
INFO:tensorflow:global_step/sec: 4.89576
INFO:tensorflow:global_step/sec: 4.89576
INFO:tensorflow:loss = 5.1261, step = 801 (20.425 sec)
INFO:tensorflow:loss = 5.1261, step = 801 (20.425 sec)
INFO:tensorflow:global_step/sec: 4.75029
INFO:tensorflow:global_step/sec: 4.75029
INFO:tensorflow:loss = 5.204457, step = 901 (21.052 sec)
INFO:tensorflow:loss = 5.204457, step = 901 (21.052 sec)
INFO:tensorflow:global_step/sec: 4.73399
INFO:tensorflow:global_step/sec: 4.73399
INFO:tensorflow:loss = 4.8827477, step = 1001 (21.117 sec)
INFO:tensorflow:loss = 4.8827477, step = 1001 (21.117 sec)
INFO:tensorflow:global_step/sec: 4.78552
INFO:tensorflow:global_step/sec: 4.78552
INFO:tensorflow:loss = 5.121219, step = 1101 (20.904 sec)
INFO:tensorflow:loss = 5.121219, step = 1101 (20.904 sec)
INFO:tensorflow:global_step/sec: 4.70474
INFO:tensorflow:global_step/sec: 4.70474
INFO:tensorflow:loss = 4.988638, step = 1201 (21.254 sec)
INFO:tensorflow:loss = 4.988638, step = 1201 (21.254 sec)
INFO:tensorflow:global_step/sec: 4.81568
INFO:tensorflow:global_step/sec: 4.81568
INFO:tensorflow:loss = 5.0044637, step = 1301 (20.760 sec)
INFO:tensorflow:loss = 5.0044637, step = 1301 (20.760 sec)
INFO:tensorflow:global_step/sec: 4.85308
INFO:tensorflow:global_step/sec: 4.85308
INFO:tensorflow:loss = 4.99024, step = 1401 (20.609 sec)
INFO:tensorflow:loss = 4.99024, step = 1401 (20.609 sec)
INFO:tensorflow:global_step/sec: 4.80463
INFO:tensorflow:global_step/sec: 4.80463
INFO:tensorflow:loss = 4.7597895, step = 1501 (20.809 sec)
INFO:tensorflow:loss = 4.7597895, step = 1501 (20.809 sec)
INFO:tensorflow:global_step/sec: 4.76081
INFO:tensorflow:global_step/sec: 4.76081
INFO:tensorflow:loss = 5.019764, step = 1601 (21.005 sec)
INFO:tensorflow:loss = 5.019764, step = 1601 (21.005 sec)
INFO:tensorflow:global_step/sec: 4.82828
INFO:tensorflow:global_step/sec: 4.82828
INFO:tensorflow:loss = 4.954829, step = 1701 (20.711 sec)
INFO:tensorflow:loss = 4.954829, step = 1701 (20.711 sec)
INFO:tensorflow:global_step/sec: 4.89424
INFO:tensorflow:global_step/sec: 4.89424
INFO:tensorflow:loss = 4.7996254, step = 1801 (20.433 sec)
INFO:tensorflow:loss = 4.7996254, step = 1801 (20.433 sec)
INFO:tensorflow:global_step/sec: 4.76087
INFO:tensorflow:global_step/sec: 4.76087
INFO:tensorflow:loss = 4.883013, step = 1901 (21.011 sec)
INFO:tensorflow:loss = 4.883013, step = 1901 (21.011 sec)
INFO:tensorflow:Saving checkpoints for 2000 into content_based_model_trained/model.ckpt.
INFO:tensorflow:Saving checkpoints for 2000 into content_based_model_trained/model.ckpt.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-05-05T15:51:44Z
INFO:tensorflow:Starting evaluation at 2020-05-05T15:51:44Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from content_based_model_trained/model.ckpt-2000
INFO:tensorflow:Restoring parameters from content_based_model_trained/model.ckpt-2000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2020-05-05-15:51:53
INFO:tensorflow:Finished evaluation at 2020-05-05-15:51:53
INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.049220674, global_step = 2000, loss = 4.907345, top_10_accuracy = 0.3178249
INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.049220674, global_step = 2000, loss = 4.907345, top_10_accuracy = 0.3178249
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2000: content_based_model_trained/model.ckpt-2000
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2000: content_based_model_trained/model.ckpt-2000
INFO:tensorflow:Loss for final step: 4.8979263.
INFO:tensorflow:Loss for final step: 4.8979263.
Out[9]:
({'accuracy': 0.049220674,
  'global_step': 2000,
  'loss': 4.907345,
  'top_10_accuracy': 0.3178249},
 [])

This takes a while to complete but in the end, I get about 30% top 10 accuracy.

Make predictions with the trained model.

With the model now trained, we can make predictions by calling the predict method on the estimator. Let's look at how our model predicts on the first five examples of the training set.
To start, we'll create a new file 'first_5.csv' which contains the first five elements of our training set. We'll also save the target values to a file 'first_5_content_ids' so we can compare our results.

In [10]:
%%bash
head -5 training_set.csv > first_5.csv
head first_5.csv
awk -F "\"*,\"*" '{print $2}' first_5.csv > first_5_content_ids
1000148716229112932,299913368,News,U4-Störung legt Wiener Frühverkehr lahm ,Yvonne Widler,574,299931241
1000148716229112932,299931241,Stars & Kultur,Regisseur Michael Haneke kritisiert Flüchtlingspolitik,,574,299913879
1000360453832106474,299925700,Lifestyle,Nach Tod von Vater: Tochter bekommt jedes Jahr Blumen,Marlene Patsalidis,574,299922662
1000360453832106474,299922662,Lifestyle,Australischer Fernsehstar rechtfertigt Sexismus mit Asperger-Syndrom,Marlene Patsalidis,574,299826775
1001846185946874596,299930679,News,Wintereinbruch naht: Erster Schnee im Osten möglich,Daniela Wahl,574,299930679

Recall, to make predictions on the trained model we pass a list of examples through the input function. Complete the code below to make predicitons on the examples contained in the "first_5.csv" file we created above.

In [11]:
output = list(estimator.predict(input_fn=read_dataset("first_5.csv", tf.estimator.ModeKeys.PREDICT)))
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from content_based_model_trained/model.ckpt-2000
INFO:tensorflow:Restoring parameters from content_based_model_trained/model.ckpt-2000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
In [12]:
import numpy as np
recommended_content_ids = [np.asscalar(d["class_names"]).decode('UTF-8') for d in output]
content_ids = open("first_5_content_ids").read().splitlines()

Finally, we map the content id back to the article title. Let's compare our model's recommendation for the first example. This can be done in BigQuery. Look through the query below and make sure it is clear what is being returned.

In [13]:
from google.cloud import bigquery
recommended_title_sql="""
#standardSQL
SELECT
(SELECT MAX(IF(index=6, value, NULL)) FROM UNNEST(hits.customDimensions)) AS title
FROM `cloud-training-demos.GA360_test.ga_sessions_sample`,   
  UNNEST(hits) AS hits
WHERE 
  # only include hits on pages
  hits.type = "PAGE"
  AND (SELECT MAX(IF(index=10, value, NULL)) FROM UNNEST(hits.customDimensions)) = \"{}\"
LIMIT 1""".format(recommended_content_ids[0])

current_title_sql="""
#standardSQL
SELECT
(SELECT MAX(IF(index=6, value, NULL)) FROM UNNEST(hits.customDimensions)) AS title
FROM `cloud-training-demos.GA360_test.ga_sessions_sample`,   
  UNNEST(hits) AS hits
WHERE 
  # only include hits on pages
  hits.type = "PAGE"
  AND (SELECT MAX(IF(index=10, value, NULL)) FROM UNNEST(hits.customDimensions)) = \"{}\"
LIMIT 1""".format(content_ids[0])
recommended_title = bigquery.Client().query(recommended_title_sql).to_dataframe()['title'].tolist()[0].encode('utf-8').strip()
current_title = bigquery.Client().query(current_title_sql).to_dataframe()['title'].tolist()[0].encode('utf-8').strip()
print("Current title: {} ".format(current_title))
print("Recommended title: {}".format(recommended_title))
Current title: U4-Störung legt Wiener Frühverkehr lahm 
Recommended title: Fahnenskandal von Mailand: Die Austria zeigt Flagge