Structured data prediction using Cloud ML Engine

This notebook illustrates:

  1. Creating datasets for Machine Learning using Dataflow
  2. Creating a model using the high-level Estimator API
  3. Training on Cloud ML Engine
  4. Deploying model
  5. Predicting with model
In [1]:
pip install --user apache-beam[gcp]==2.16.0
Collecting apache-beam[gcp]==2.16.0
  Downloading https://files.pythonhosted.org/packages/0f/15/a8065042472311383f34d94fe3ff611cc7ab092b0bf502ad097acb7406e3/apache_beam-2.16.0-cp35-cp35m-manylinux1_x86_64.whl (2.9MB)
    100% |████████████████████████████████| 2.9MB 454kB/s eta 0:00:01
Collecting future<1.0.0,>=0.16.0 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB)
    100% |████████████████████████████████| 829kB 1.6MB/s eta 0:00:01
Collecting httplib2<=0.12.0,>=0.8 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/ce/ed/803905d670b52fa0edfdd135337e545b4496c2ab3a222f1449b7256eb99f/httplib2-0.12.0.tar.gz (218kB)
    100% |████████████████████████████████| 225kB 5.5MB/s eta 0:00:01
Collecting mock<3.0.0,>=1.0.1 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 9.9MB/s eta 0:00:01
Collecting oauth2client<4,>=2.0.1 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/c0/7b/bc893e35d6ca46a72faa4b9eaac25c687ce60e1fbe978993fe2de1b0ff0d/oauth2client-3.0.0.tar.gz (77kB)
    100% |████████████████████████████████| 81kB 10.3MB/s ta 0:00:01
Collecting fastavro<0.22,>=0.21.4 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/ac/7d/e63a1ba78326e42a69bda88b1fcfca22ddd773c4cc51ae85b3b869abcff2/fastavro-0.21.24-cp35-cp35m-manylinux1_x86_64.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 1.1MB/s eta 0:00:01
Collecting pytz>=2018.3 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/4f/a4/879454d49688e2fad93e59d7d4efda580b783c745fd2ec2a3adf87b0808d/pytz-2020.1-py2.py3-none-any.whl (510kB)
    100% |████████████████████████████████| 512kB 2.8MB/s eta 0:00:01
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)
    100% |████████████████████████████████| 276kB 4.8MB/s eta 0:00:01
Collecting crcmod<2.0,>=1.7 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/6b/b0/e595ce2a2527e169c3bcd6c33d2473c1918e0b7f6826a043ca1245dd4e5b/crcmod-1.7.tar.gz (89kB)
    100% |████████████████████████████████| 92kB 10.7MB/s ta 0:00:01
Collecting pymongo<4.0.0,>=3.8.0 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/5b/08/c0e0f1dff3a996f5ef4ba0cedf5d4c900748bbe5f1a811e78880678046d2/pymongo-3.10.1-cp35-cp35m-manylinux1_x86_64.whl (450kB)
    100% |████████████████████████████████| 460kB 2.9MB/s eta 0:00:01
Collecting python-dateutil<3,>=2.8.0 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
    100% |████████████████████████████████| 235kB 5.6MB/s eta 0:00:01
Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/5a/80/acd1455bea0a9fcdc60a748a97dcbb3ff624726fb90987a0fc1c19e7a5a5/avro-python3-1.9.2.1.tar.gz
Collecting pydot<2,>=1.2.0 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/33/d1/b1479a770f66d962f545c2101630ce1d5592d90cb4f083d38862e93d16d2/pydot-1.4.1-py2.py3-none-any.whl
Collecting grpcio<2,>=1.12.1 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/cf/7a/9744998129fce7e29c5f2d8b0f545913b7383e65d8366fc0ae98d11936af/grpcio-1.28.1.tar.gz (19.5MB)
    100% |████████████████████████████████| 19.5MB 67kB/s  eta 0:00:01
Collecting dill<0.3.1,>=0.3.0 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/39/7a/70803635c850e351257029089d38748516a280864c97cbc73087afef6d51/dill-0.3.0.tar.gz (151kB)
    100% |████████████████████████████████| 153kB 7.8MB/s eta 0:00:01
Collecting protobuf<4,>=3.5.0.post1 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/ef/56/10999d97e62eba72a12ebd147d806fa702ab82a620e28f6bc3e5d2c59e34/protobuf-3.11.3-cp35-cp35m-manylinux1_x86_64.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 1.0MB/s eta 0:00:01
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/82/39/2c0879b1bcfd1f6ad078eb210d09dbce21072386a3997074ee91e60ddc5a/hdfs-2.5.8.tar.gz (41kB)
    100% |████████████████████████████████| 51kB 9.9MB/s eta 0:00:01
Collecting pyarrow<0.15.0,>=0.11.1; python_version >= "3.0" or platform_system != "Windows" (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/02/3a/6b9474507e6abbd2bbe715627fec991e3c71f5ea71daff1ebddeddab0208/pyarrow-0.14.1-cp35-cp35m-manylinux1_x86_64.whl (58.2MB)
    100% |████████████████████████████████| 58.2MB 20kB/s  eta 0:00:01
Collecting google-cloud-bigtable<1.1.0,>=0.31.1; extra == "gcp" (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/95/af/0ef7d097a1d5ad0c843867600e86de915e8ab8864740f49a4636cfb51af6/google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232kB)
    100% |████████████████████████████████| 235kB 5.4MB/s eta 0:00:01
Collecting google-cloud-datastore<1.8.0,>=1.7.1; extra == "gcp" (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/d0/aa/29cbcf8cf7d08ce2d55b9dce858f7c632b434cb6451bed17cb4275804217/google_cloud_datastore-1.7.4-py2.py3-none-any.whl (82kB)
    100% |████████████████████████████████| 92kB 10.5MB/s ta 0:00:01
Collecting cachetools<4,>=3.1.0; extra == "gcp" (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/2f/a6/30b0a0bef12283e83e58c1d6e7b5aabc7acfc4110df81a4471655d33e704/cachetools-3.1.1-py2.py3-none-any.whl
Collecting google-cloud-bigquery<1.18.0,>=1.6.0; extra == "gcp" (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/a4/96/1b9cf1d43869c47a205aad411dac7c3040df6093d63c39273fa4d4c45da7/google_cloud_bigquery-1.17.1-py2.py3-none-any.whl (142kB)
    100% |████████████████████████████████| 143kB 8.1MB/s eta 0:00:01
Collecting google-cloud-pubsub<1.1.0,>=0.39.0; extra == "gcp" (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/d3/91/07a82945a7396ea34debafd476724bb5fc267c292790fdf2138c693f95c5/google_cloud_pubsub-1.0.2-py2.py3-none-any.whl (118kB)
    100% |████████████████████████████████| 122kB 9.3MB/s eta 0:00:01
Collecting google-cloud-core<2,>=0.28.1; extra == "gcp" (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/89/3c/8a7531839028c9690e6d14c650521f3bbaf26e53baaeb2784b8c3eb2fb97/google_cloud_core-1.3.0-py2.py3-none-any.whl
Collecting google-apitools<0.5.29,>=0.5.28; extra == "gcp" (from apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/7f/32/df3e36fd705a00092f1ffa9f41ce1df8dcb594ae313d239b87861a41fc2e/google-apitools-0.5.28.tar.gz (172kB)
    100% |████████████████████████████████| 174kB 6.5MB/s eta 0:00:01
Collecting six>=1.9 (from mock<3.0.0,>=1.0.1->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/96/ba/aa953a11ec014b23df057ecdbc922fdb40ca8463466b1193f3367d2711a6/pbr-5.4.5-py2.py3-none-any.whl (110kB)
    100% |████████████████████████████████| 112kB 9.3MB/s eta 0:00:01
Collecting pyasn1>=0.1.7 (from oauth2client<4,>=2.0.1->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77kB)
    100% |████████████████████████████████| 81kB 9.8MB/s eta 0:00:01
Collecting pyasn1-modules>=0.0.5 (from oauth2client<4,>=2.0.1->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl (155kB)
    100% |████████████████████████████████| 163kB 7.0MB/s eta 0:00:01
Collecting rsa>=3.1.4 (from oauth2client<4,>=2.0.1->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl
Collecting pyparsing>=2.1.4 (from pydot<2,>=1.2.0->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
    100% |████████████████████████████████| 71kB 9.3MB/s eta 0:00:01
Collecting setuptools (from protobuf<4,>=3.5.0.post1->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/a0/df/635cdb901ee4a8a42ec68e480c49f85f4c59e8816effbf57d9e6ee8b3588/setuptools-46.1.3-py3-none-any.whl (582kB)
    100% |████████████████████████████████| 583kB 2.2MB/s eta 0:00:01
Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz
Collecting requests>=2.7.0 (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 10.0MB/s ta 0:00:01
Collecting numpy>=1.14 (from pyarrow<0.15.0,>=0.11.1; python_version >= "3.0" or platform_system != "Windows"->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/38/92/fa5295d9755c7876cb8490eab866e1780154033fa45978d9cf74ffbd4c68/numpy-1.18.4-cp35-cp35m-manylinux1_x86_64.whl (20.0MB)
    100% |████████████████████████████████| 20.0MB 64kB/s  eta 0:00:01
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0 (from google-cloud-bigtable<1.1.0,>=0.31.1; extra == "gcp"->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/4c/b9/c0dd70bcdf06a43d1e21f387448e7997e0ce91f10d0fbee359af4cde1571/google_api_core-1.17.0-py2.py3-none-any.whl (70kB)
    100% |████████████████████████████████| 71kB 9.3MB/s eta 0:00:01
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3 (from google-cloud-bigtable<1.1.0,>=0.31.1; extra == "gcp"->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/65/19/2060c8faa325fddc09aa67af98ffcb6813f39a0ad805679fa64815362b3a/grpc-google-iam-v1-0.12.3.tar.gz
Collecting google-resumable-media<0.5.0dev,>=0.3.1 (from google-cloud-bigquery<1.18.0,>=1.6.0; extra == "gcp"->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/96/d7/b29a41b01b854480891dfc408211ffb0cc7a2a3d5f15a3b6740ec18c845b/google_resumable_media-0.4.1-py2.py3-none-any.whl
Collecting fasteners>=0.14 (from google-apitools<0.5.29,>=0.5.28; extra == "gcp"->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/18/bd/55eb2d6397b9c0e263af9d091ebdb756b15756029b3cededf6461481bc63/fasteners-0.15-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/57/2b/26e37a4b034800c960a00c4e1b3d9ca5d7014e983e6e729e33ea2f36426c/certifi-2020.4.5.1-py2.py3-none-any.whl (157kB)
    100% |████████████████████████████████| 163kB 7.4MB/s eta 0:00:01
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/e1/e5/df302e8017440f111c11cc41a6b432838672f5a70aa29227bf58149dc72f/urllib3-1.25.9-py2.py3-none-any.whl (126kB)
    100% |████████████████████████████████| 133kB 9.1MB/s eta 0:00:01
Collecting chardet<4,>=3.0.2 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 8.3MB/s eta 0:00:01
Collecting idna<3,>=2.5 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/89/e3/afebe61c546d18fb1709a61bee788254b40e736cff7271c7de5de2dc4128/idna-2.9-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 9.0MB/s eta 0:00:01
Collecting google-auth<2.0dev,>=1.14.0 (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-bigtable<1.1.0,>=0.31.1; extra == "gcp"->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/d2/f8/1623d69e5de22e499b68a0cb5e5d02cd6a2843e55acc19f314f48fe04299/google_auth-1.14.1-py2.py3-none-any.whl (89kB)
    100% |████████████████████████████████| 92kB 9.8MB/s eta 0:00:01
Collecting googleapis-common-protos<2.0dev,>=1.6.0 (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-bigtable<1.1.0,>=0.31.1; extra == "gcp"->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/05/46/168fd780f594a4d61122f7f3dc0561686084319ad73b4febbf02ae8b32cf/googleapis-common-protos-1.51.0.tar.gz
Collecting monotonic>=0.1 (from fasteners>=0.14->google-apitools<0.5.29,>=0.5.28; extra == "gcp"->apache-beam[gcp]==2.16.0)
  Downloading https://files.pythonhosted.org/packages/ac/aa/063eca6a416f397bd99552c534c6d11d57f58f2e94c14780f3bbf818c4cf/monotonic-1.5-py2.py3-none-any.whl
Building wheels for collected packages: future, httplib2, oauth2client, pyyaml, crcmod, avro-python3, grpcio, dill, hdfs, google-apitools, docopt, grpc-google-iam-v1, googleapis-common-protos
  Running setup.py bdist_wheel for future ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e
  Running setup.py bdist_wheel for httplib2 ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/6d/41/4b/2b369d6e2b7eaebcdd423516d3fb659c7658c16a2be8fd04ec
  Running setup.py bdist_wheel for oauth2client ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/48/f7/87/b932f09c6335dbcf45d916937105a372ab14f353a9ca431d7d
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f
  Running setup.py bdist_wheel for crcmod ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/50/24/4d/4580ca4a299f1ad6fd63443e6e584cb21e9a07988e4aa8daac
  Running setup.py bdist_wheel for avro-python3 ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/5a/d3/be/86620c9dd3fca68986c33b9c616510289fc0abb81ec9aa70bd
  Running setup.py bdist_wheel for grpcio ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd10c33
  Running setup.py bdist_wheel for dill ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/c9/de/a4/a91eec4eea652104d8c81b633f32ead5eb57d1b294eab24167
  Running setup.py bdist_wheel for hdfs ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/fe/a7/05/23e3699975fc20f8a30e00ac1e515ab8c61168e982abe4ce70
  Running setup.py bdist_wheel for google-apitools ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/d6/c2/92/837e8a4d649a209dff85b38d7fbb576b4b480738be70865f29
  Running setup.py bdist_wheel for docopt ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e
  Running setup.py bdist_wheel for grpc-google-iam-v1 ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/de/3a/83/77a1e18e1a8757186df834b86ce6800120ac9c79cd8ca4091b
  Running setup.py bdist_wheel for googleapis-common-protos ... done
  Stored in directory: /home/jupyter/.cache/pip/wheels/2c/f9/7f/6eb87e636072bf467e25348bbeb96849333e6a080dca78f706
Successfully built future httplib2 oauth2client pyyaml crcmod avro-python3 grpcio dill hdfs google-apitools docopt grpc-google-iam-v1 googleapis-common-protos
Installing collected packages: future, httplib2, six, pbr, mock, pyasn1, pyasn1-modules, rsa, oauth2client, fastavro, pytz, pyyaml, crcmod, pymongo, python-dateutil, avro-python3, pyparsing, pydot, grpcio, dill, setuptools, protobuf, docopt, certifi, urllib3, chardet, idna, requests, hdfs, numpy, pyarrow, cachetools, google-auth, googleapis-common-protos, google-api-core, grpc-google-iam-v1, google-cloud-core, google-cloud-bigtable, google-cloud-datastore, google-resumable-media, google-cloud-bigquery, google-cloud-pubsub, monotonic, fasteners, google-apitools, apache-beam
Successfully installed apache-beam-2.16.0 avro-python3-1.9.2.1 cachetools-3.1.1 certifi-2020.4.5.1 chardet-3.0.4 crcmod-1.7 dill-0.3.0 docopt-0.6.2 fastavro-0.21.24 fasteners-0.15 future-0.18.2 google-api-core-1.17.0 google-apitools-0.5.28 google-auth-1.14.1 google-cloud-bigquery-1.17.1 google-cloud-bigtable-1.0.0 google-cloud-core-1.3.0 google-cloud-datastore-1.7.4 google-cloud-pubsub-1.0.2 google-resumable-media-0.4.1 googleapis-common-protos-1.51.0 grpc-google-iam-v1-0.12.3 grpcio-1.28.1 hdfs-2.5.8 httplib2-0.12.0 idna-2.9 mock-2.0.0 monotonic-1.5 numpy-1.18.4 oauth2client-3.0.0 pbr-5.4.5 protobuf-3.11.3 pyarrow-0.14.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydot-1.4.1 pymongo-3.10.1 pyparsing-2.4.7 python-dateutil-2.8.1 pytz-2020.1 pyyaml-3.13 requests-2.23.0 rsa-4.0 setuptools-46.1.3 six-1.14.0 urllib3-1.25.9
Note: you may need to restart the kernel to use updated packages.

Run the command again if you are getting oauth2client error. Restart the kernel before proceeding further.

In [2]:
# change these to try this notebook out
BUCKET = 'my-bucket' # Replace with the your bucket name
PROJECT = 'my-project' # Replace with your project-id
REGION = 'us-central1'
In [3]:
import os
os.environ['BUCKET'] = BUCKET
os.environ['PROJECT'] = PROJECT
os.environ['REGION'] = REGION
In [4]:
%%bash
gcloud config set project $PROJECT
Updated property [core/project].
In [5]:
%%bash
if ! gsutil ls | grep -q gs://${BUCKET}/; then
  gsutil mb -l ${REGION} gs://${BUCKET}
fi
gsutil -m cp -R gs://cloud-training-demos/babyweight gs://${BUCKET}
Copying gs://cloud-training-demos/babyweight/hyperparam/1/checkpoint...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/eval/events.out.tfevents.1529347627.cmle-training-master-ab97329ccf-0-xfhjt...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/export/exporter/1529347629/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/events.out.tfevents.1529347394.cmle-training-master-ab97329ccf-0-xfhjt...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/export/exporter/1529347629/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/export/exporter/1529347629/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/export/exporter/1529347926/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/export/exporter/1529348269/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/graph.pbtxt...        
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-141975.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/export/exporter/1529348269/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/export/exporter/1529347926/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-141975.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/export/exporter/1529348269/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-141975.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-141975.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/export/exporter/1529347926/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-295591.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-141975.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-295591.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-295591.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-295591.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-295591.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-317465.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-317465.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-317465.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-317465.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/1/model.ckpt-317465.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/checkpoint...        
Copying gs://cloud-training-demos/babyweight/hyperparam/10/eval/events.out.tfevents.1529349910.cmle-training-master-3034bab816-0-sjbrw...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/events.out.tfevents.1529349469.cmle-training-master-3034bab816-0-sjbrw...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/export/exporter/1529349912/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/export/exporter/1529349912/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/export/exporter/1529349912/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/export/exporter/1529350256/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/export/exporter/1529350256/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/export/exporter/1529350256/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/graph.pbtxt...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-25.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-25.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-25.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-25.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-25.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-36177.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-36177.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-36177.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-36177.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-36177.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-41931.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-41931.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-41931.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-41931.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/10/model.ckpt-41931.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/checkpoint...        
Copying gs://cloud-training-demos/babyweight/hyperparam/11/eval/events.out.tfevents.1529350196.cmle-training-master-147ed9d6ac-0-xxzz7...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/events.out.tfevents.1529350033.cmle-training-master-147ed9d6ac-0-xxzz7...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/export/exporter/1529350198/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/export/exporter/1529350198/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/export/exporter/1529350198/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/export/exporter/1529350524/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/export/exporter/1529350524/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/export/exporter/1529350524/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/graph.pbtxt...       
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-1.meta... 
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-35698.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-35698.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-35698.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-35698.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-35698.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-39142.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-39142.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-39142.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-39142.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/11/model.ckpt-39142.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/checkpoint...        
Copying gs://cloud-training-demos/babyweight/hyperparam/12/eval/events.out.tfevents.1529350215.cmle-training-master-91e18bf984-0-glpz6...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/events.out.tfevents.1529350059.cmle-training-master-91e18bf984-0-glpz6...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/export/exporter/1529350216/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/export/exporter/1529350216/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/export/exporter/1529350216/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/export/exporter/1529350499/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/export/exporter/1529350499/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/export/exporter/1529350499/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/graph.pbtxt...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-1.meta... 
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-52777.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-52777.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-52777.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-52777.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/12/model.ckpt-52777.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/checkpoint...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/eval/events.out.tfevents.1529350568.cmle-training-master-2951e6ce4f-0-cjrmj...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/events.out.tfevents.1529350538.cmle-training-master-2951e6ce4f-0-cjrmj...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/export/exporter/1529350570/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/export/exporter/1529350570/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/export/exporter/1529350570/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/export/exporter/1529350871/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/export/exporter/1529350871/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/export/exporter/1529350871/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/export/exporter/1529350905/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/export/exporter/1529350905/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/export/exporter/1529350905/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/graph.pbtxt...       
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-1.meta... 
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-42291.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-42291.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-42291.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-42291.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-42291.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-43297.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-43297.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-43297.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-43297.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/13/model.ckpt-43297.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/checkpoint...        
Copying gs://cloud-training-demos/babyweight/hyperparam/14/eval/events.out.tfevents.1529350566.cmle-training-master-58d1f4795d-0-8mrg6...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/events.out.tfevents.1529350542.cmle-training-master-58d1f4795d-0-8mrg6...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/export/exporter/1529350568/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/export/exporter/1529350568/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/export/exporter/1529350568/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/graph.pbtxt...       
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-1.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-39142.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-39142.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-39142.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-39142.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-39142_temp_34004248bb4b436d9a67b7e638b0bbb3/part-00000-of-00003.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-39142_temp_34004248bb4b436d9a67b7e638b0bbb3/part-00001-of-00003.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/14/model.ckpt-39142_temp_34004248bb4b436d9a67b7e638b0bbb3/part-00002-of-00003.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/checkpoint...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/eval/events.out.tfevents.1529350856.cmle-training-master-c4e81fbb25-0-sz655...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/events.out.tfevents.1529350831.cmle-training-master-c4e81fbb25-0-sz655...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/export/exporter/1529350858/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/export/exporter/1529350858/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/export/exporter/1529350858/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/export/exporter/1529351128/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/export/exporter/1529351128/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/export/exporter/1529351128/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/graph.pbtxt...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-1.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-39142.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-39142.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-39142.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-39142.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/15/model.ckpt-39142.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/checkpoint...        
Copying gs://cloud-training-demos/babyweight/hyperparam/16/eval/events.out.tfevents.1529351171.cmle-training-master-3523a35558-0-92n89...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/events.out.tfevents.1529350997.cmle-training-master-3523a35558-0-92n89...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/export/exporter/1529351172/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/export/exporter/1529351172/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/export/exporter/1529351172/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/export/exporter/1529351482/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/export/exporter/1529351482/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/export/exporter/1529351482/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/graph.pbtxt...       
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-1.meta... 
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-37295.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-37295.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-37295.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-37295.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-37295.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-39146.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-39146.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-39146.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-39146.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/16/model.ckpt-39146.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/checkpoint...        
Copying gs://cloud-training-demos/babyweight/hyperparam/17/eval/events.out.tfevents.1529351354.cmle-training-master-dc3a8c26ae-0-v6sld...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/events.out.tfevents.1529351326.cmle-training-master-dc3a8c26ae-0-v6sld...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/export/exporter/1529351355/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/export/exporter/1529351355/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/export/exporter/1529351355/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/export/exporter/1529351625/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/export/exporter/1529351625/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/export/exporter/1529351625/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/graph.pbtxt...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-1.meta... 
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-39300.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-39300.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-39300.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-39300.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/17/model.ckpt-39300.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/checkpoint...        
Copying gs://cloud-training-demos/babyweight/hyperparam/18/eval/events.out.tfevents.1529351495.cmle-training-master-d1e5f6ad31-0-dbw9g...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/events.out.tfevents.1529351469.cmle-training-master-d1e5f6ad31-0-dbw9g...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/export/exporter/1529351496/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/export/exporter/1529351496/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/export/exporter/1529351496/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/export/exporter/1529351787/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/export/exporter/1529351787/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/export/exporter/1529351787/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/graph.pbtxt...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-1.meta... 
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-39218.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-39218.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-39218.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-39218.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/18/model.ckpt-39218.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/checkpoint...        
Copying gs://cloud-training-demos/babyweight/hyperparam/19/eval/events.out.tfevents.1529351745.cmle-training-master-67c24f2848-0-sqcqw...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/events.out.tfevents.1529351716.cmle-training-master-67c24f2848-0-sqcqw...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/export/exporter/1529351747/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/export/exporter/1529351747/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/export/exporter/1529351747/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/export/exporter/1529352047/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/export/exporter/1529352047/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/export/exporter/1529352047/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/export/exporter/1529352079/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/export/exporter/1529352079/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/export/exporter/1529352079/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/graph.pbtxt...       
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-1.meta... 
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-39733.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-39733.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-39733.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-39733.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-39733.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-40167.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-40167.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-40167.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-40167.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/19/model.ckpt-40167.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/checkpoint...         
Copying gs://cloud-training-demos/babyweight/hyperparam/2/eval/events.out.tfevents.1529347521.cmle-training-master-562372026f-0-vkqw8...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/events.out.tfevents.1529347361.cmle-training-master-562372026f-0-vkqw8...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/export/exporter/1529347523/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/export/exporter/1529347523/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/export/exporter/1529347523/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/export/exporter/1529347809/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/export/exporter/1529347809/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/export/exporter/1529347809/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/graph.pbtxt...        
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-22.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-22.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-22.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-22.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-22.meta... 
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-41414.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-41414.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-41414.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-41414.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/2/model.ckpt-41414.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/checkpoint...        
Copying gs://cloud-training-demos/babyweight/hyperparam/20/eval/events.out.tfevents.1529352130.cmle-training-master-9194a02df3-0-2jhm6...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/events.out.tfevents.1529351968.cmle-training-master-9194a02df3-0-2jhm6...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/export/exporter/1529352132/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/export/exporter/1529352132/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/export/exporter/1529352132/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/export/exporter/1529352442/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/export/exporter/1529352442/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/export/exporter/1529352442/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/graph.pbtxt...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-14.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-14.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-14.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-14.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-37770.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-14.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-37770.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-37770.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-37770.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-37770.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-39222.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-39222.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-39222.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-39222.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/20/model.ckpt-39222.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/checkpoint...         
Copying gs://cloud-training-demos/babyweight/hyperparam/3/eval/events.out.tfevents.1529347749.cmle-training-master-32c4950845-0-l8mtk...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/events.out.tfevents.1529347399.cmle-training-master-32c4950845-0-l8mtk...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529347751/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529347751/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529347751/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348048/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348048/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348048/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348350/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348350/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348350/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348648/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348648/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348648/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348952/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348952/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529348952/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529349251/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529349251/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/export/exporter/1529349251/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1013057.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/graph.pbtxt...        
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1013057.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1013057.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1013057.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1013057.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1269118.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1269118.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1269118.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1269118.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-1269118.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-754997.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-754997.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-754997.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-754997.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/3/model.ckpt-754997.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/4/events.out.tfevents.1529347396.cmle-training-master-f54adf8366-0-9vbn8...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/checkpoint...         
Copying gs://cloud-training-demos/babyweight/hyperparam/5/eval/events.out.tfevents.1529347752.cmle-training-master-a7c3435684-0-k49qb...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/events.out.tfevents.1529347396.cmle-training-master-a7c3435684-0-k49qb...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529347753/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529347753/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529347753/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348051/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348051/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348051/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348350/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348350/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348350/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348651/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348651/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348651/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348952/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348952/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/export/exporter/1529348952/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/graph.pbtxt...        
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-572586.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-572586.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-572586.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-572586.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-572586.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-767695.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-767695.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-767695.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-767695.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-767695.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-959128.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-959128.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-959128.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-959128.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/5/model.ckpt-959128.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/checkpoint...         
Copying gs://cloud-training-demos/babyweight/hyperparam/6/eval/events.out.tfevents.1529348214.cmle-training-master-ec16a0895d-0-xtx5j...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/events.out.tfevents.1529348187.cmle-training-master-ec16a0895d-0-xtx5j...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/export/exporter/1529348216/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/export/exporter/1529348216/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/export/exporter/1529348216/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/export/exporter/1529348496/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/export/exporter/1529348496/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/export/exporter/1529348496/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/graph.pbtxt...        
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-1.index... 
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-1.meta...  
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-41935.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-41935.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-41935.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-41935.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/6/model.ckpt-41935.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/checkpoint...         
Copying gs://cloud-training-demos/babyweight/hyperparam/7/eval/events.out.tfevents.1529348617.cmle-training-master-34de34160b-0-kzg6l...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/events.out.tfevents.1529348592.cmle-training-master-34de34160b-0-kzg6l...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/export/exporter/1529348619/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/export/exporter/1529348619/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/export/exporter/1529348619/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/export/exporter/1529348918/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/export/exporter/1529348918/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/export/exporter/1529348918/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/export/exporter/1529349139/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/export/exporter/1529349139/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/export/exporter/1529349139/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/graph.pbtxt...        
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-1.index... 
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-1.meta...  
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-179951.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-179951.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-179951.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-179951.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-179951.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-317466.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-317466.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-317466.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-317466.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/7/model.ckpt-317466.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/checkpoint...         
Copying gs://cloud-training-demos/babyweight/hyperparam/8/eval/events.out.tfevents.1529348857.cmle-training-master-4cc903f03c-0-wgg9f...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/events.out.tfevents.1529348822.cmle-training-master-4cc903f03c-0-wgg9f...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/export/exporter/1529348859/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/export/exporter/1529348859/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/export/exporter/1529348859/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/export/exporter/1529349124/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/export/exporter/1529349124/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/export/exporter/1529349124/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/graph.pbtxt...        
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-1.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-40166.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-40166.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-40166.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-40166.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/8/model.ckpt-40166.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/checkpoint...         
Copying gs://cloud-training-demos/babyweight/hyperparam/9/eval/events.out.tfevents.1529349999.cmle-training-master-48340617c9-0-n4cg7...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/events.out.tfevents.1529349470.cmle-training-master-48340617c9-0-n4cg7...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/export/exporter/1529350000/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/export/exporter/1529350000/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/export/exporter/1529350000/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/export/exporter/1529350336/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/export/exporter/1529350336/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/export/exporter/1529350336/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/graph.pbtxt...        
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-1.index... 
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-1.meta...  
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-35917.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-35917.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-35917.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-35917.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-35917.meta...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-40655.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-40655.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-40655.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-40655.index...
Copying gs://cloud-training-demos/babyweight/hyperparam/9/model.ckpt-40655.meta...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00000-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00001-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00002-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00003-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00004-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00006-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00005-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00007-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00008-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00009-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00010-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/eval.csv-00011-of-00012 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00000-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00001-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00002-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00003-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00004-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00005-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00006-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00007-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00008-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00009-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00010-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00011-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00012-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00013-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00014-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00015-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00016-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00017-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00018-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00019-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00020-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00021-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00022-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00023-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00024-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00025-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00026-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00027-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00028-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00029-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00030-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00031-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00032-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00033-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00034-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00035-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00036-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00037-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00038-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00039-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00041-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00040-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/preproc/train.csv-00042-of-00043 [Content-Type=text/plain]...
Copying gs://cloud-training-demos/babyweight/trained_model/checkpoint...        
Copying gs://cloud-training-demos/babyweight/trained_model/eval/events.out.tfevents.1529348264.cmle-training-master-a137ac0fff-0-9q8r4...
Copying gs://cloud-training-demos/babyweight/trained_model/events.out.tfevents.1529347276.cmle-training-master-a137ac0fff-0-9q8r4...
Copying gs://cloud-training-demos/babyweight/trained_model/export/exporter/1529355466/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/trained_model/export/exporter/1529355466/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/trained_model/export/exporter/1529355466/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/trained_model/graph.pbtxt...       
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-342784.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-342784.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-342784.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-342784.index...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-342784.meta...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-376661.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-376661.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-376661.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-376661.index...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-376661.meta...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-390628.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-390628.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-390628.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-390628.index...
Copying gs://cloud-training-demos/babyweight/trained_model/model.ckpt-390628.meta...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/eval/events.out.tfevents.1529348486.cmle-training-master-d20efff6e7-0-gtx4t...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/checkpoint...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/events.out.tfevents.1529347278.cmle-training-master-d20efff6e7-0-gtx4t...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/export/exporter/1529348487/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/export/exporter/1529348487/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/export/exporter/1529348487/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/export/exporter/1529349318/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/export/exporter/1529349318/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/export/exporter/1529349318/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/export/exporter/1529350150/saved_model.pb...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/export/exporter/1529350150/variables/variables.data-00000-of-00001...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/export/exporter/1529350150/variables/variables.index...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/graph.pbtxt...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-1.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-1.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-1.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-1.index...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-1.meta...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-483973.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-483973.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-483973.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-483973.index...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-483973.meta...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-571432.data-00000-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-571432.data-00001-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-571432.data-00002-of-00003...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-571432.index...
Copying gs://cloud-training-demos/babyweight/trained_model_tuned/model.ckpt-571432.meta...
\ [573/573 files][  6.1 GiB/  6.1 GiB] 100% Done                                
Operation completed over 573 objects/6.1 GiB.                                    

A variety of factors seem to play a part in the baby's weight. Male babies are heavier on average than female babies. Teenaged and older moms tend to have lower-weight babies. Twins, triplets, etc. are lower weight than single births. Preemies weigh in lower as do babies born to single moms. In addition, it is important to check whether you have enough data (number of babies) for each input value. Otherwise, the model prediction against input values that doesn't have enough data may not be reliable.

In the rest of this notebook, we will use machine learning to combine all of these factors to come up with a prediction of a baby's weight.

Create ML dataset using Dataflow

We can use Cloud Dataflow to read in the BigQuery data and write it out as CSV files.

Instead of using Beam/Dataflow, I had three other options:

  1. Use Cloud Dataprep to visually author a Dataflow pipeline. Cloud Dataprep also allows me to explore the data, so we could have avoided much of the handcoding of Python/Seaborn calls above as well!

  2. Read from BigQuery directly using TensorFlow.

  3. Use the BigQuery console (http://bigquery.cloud.google.com) to run a Query and save the result as a CSV file. For larger datasets, you may have to select the option to "allow large results" and save the result into a CSV file on Google Cloud Storage. </ol>

    However, in this case, I want to do some preprocessing, modifying data so that we can simulate what is known if no ultrasound has been performed. If I didn't need preprocessing, I could have used the web console. Also, I prefer to script it out rather than run queries on the user interface, so I am using Cloud Dataflow for the preprocessing.

    Note that I have set in_test_mode=True in the following code -- this will run the code locally on a small subset of the data -- the full results were copied into your bucket using the following code (in the previous cell):

    gsutil -m cp -R gs://cloud-training-demos/babyweight gs://${BUCKET}
    
    If you are running in your own project, change in_test_mode=False; after you launch this, the notebook will appear to be hung. Go to the GCP webconsole to the Dataflow section and monitor the running job. It took about 30 minutes for me with autoscaling to 15 workers -- Qwiklabs accounts won't be able scale to that level, and so doing it on Qwiklabs would have taken 3-4 hours. Hence, the short-cut.

In [6]:
import apache_beam as beam
import datetime

def to_csv(rowdict):
    # pull columns from BQ and create a line
    import hashlib
    import copy
    CSV_COLUMNS = 'weight_pounds,is_male,mother_age,plurality,gestation_weeks'.split(',')
    
    # create synthetic data where we assume that no ultrasound has been performed
    # and so we don't know sex of the baby. Let's assume that we can tell the difference
    # between single and multiple, but that the errors rates in determining exact number
    # is difficult in the absence of an ultrasound.
    no_ultrasound = copy.deepcopy(rowdict)
    w_ultrasound = copy.deepcopy(rowdict)

    no_ultrasound['is_male'] = 'Unknown'
    if rowdict['plurality'] > 1:
      no_ultrasound['plurality'] = 'Multiple(2+)'
    else:
      no_ultrasound['plurality'] = 'Single(1)'
      
    # Change the plurality column to strings
    w_ultrasound['plurality'] = ['Single(1)', 'Twins(2)', 'Triplets(3)', 'Quadruplets(4)', 'Quintuplets(5)'][rowdict['plurality']-1]
    
    # Write out two rows for each input row, one with ultrasound and one without
    for result in [no_ultrasound, w_ultrasound]:
      data = ','.join([str(result[k]) if k in result else 'None' for k in CSV_COLUMNS])
      key = hashlib.sha224(data.encode('utf-8')).hexdigest()  # hash the columns to form a key
      yield str('{},{}'.format(data, key))
  
def preprocess(in_test_mode):
    job_name = 'preprocess-babyweight-features' + '-' + datetime.datetime.now().strftime('%y%m%d-%H%M%S')
    
    if in_test_mode:
        print ('Launching local job ... hang on')
        OUTPUT_DIR = './preproc'
    else:
        print ('Launching Dataflow job {} ... hang on'.format(job_name))
        OUTPUT_DIR = 'gs://{0}/babyweight/preproc/'.format(BUCKET)
    
    options = {
        'staging_location': os.path.join(OUTPUT_DIR, 'tmp', 'staging'),
        'temp_location': os.path.join(OUTPUT_DIR, 'tmp'),
        'job_name': job_name,
        'project': PROJECT,
        'teardown_policy': 'TEARDOWN_ALWAYS',
        'no_save_main_session': True
    }
    opts = beam.pipeline.PipelineOptions(flags=[], **options)
    if in_test_mode:
        RUNNER = 'DirectRunner'
    else:
        RUNNER = 'DataflowRunner'
    p = beam.Pipeline(RUNNER, options=opts)
    query = """
SELECT
  weight_pounds,
  is_male,
  mother_age,
  plurality,
  gestation_weeks,
  FARM_FINGERPRINT(CONCAT(CAST(YEAR AS STRING), CAST(month AS STRING))) AS hashmonth
FROM
  publicdata.samples.natality
WHERE year > 2000
AND weight_pounds > 0
AND mother_age > 0
AND plurality > 0
AND gestation_weeks > 0
AND month > 0
    """
  
    if in_test_mode:
        query = query + ' LIMIT 100' 
  
    for step in ['train', 'eval']:
        if step == 'train':
            selquery = 'SELECT * FROM ({}) WHERE ABS(MOD(hashmonth, 4)) < 3'.format(query)
        else:
            selquery = 'SELECT * FROM ({}) WHERE ABS(MOD(hashmonth, 4)) = 3'.format(query)

        (p 
         | '{}_read'.format(step) >> beam.io.Read(beam.io.BigQuerySource(query=selquery, use_standard_sql=True))
         | '{}_csv'.format(step) >> beam.FlatMap(to_csv)
         | '{}_out'.format(step) >> beam.io.Write(beam.io.WriteToText(os.path.join(OUTPUT_DIR, '{}.csv'.format(step))))
        )
 
    job = p.run()
  
preprocess(in_test_mode=True)
Launching local job ... hang on
WARNING:root:The DirectPipelineResult is being garbage-collected while the DirectRunner is still running the corresponding pipeline. This may lead to incomplete execution of the pipeline if the main thread exits before pipeline completion. Consider using result.wait_until_finish() to wait for completion of pipeline execution.
WARNING:root:Dataset qwiklabs-gcp-01-c5358f6c8e9a:temp_dataset_fc544223bb0343fab640e78916e80634 does not exist so we will create it as temporary with location=US
WARNING:root:Dataset qwiklabs-gcp-01-c5358f6c8e9a:temp_dataset_26717b8380e646db98dd5231b74a9cf4 does not exist so we will create it as temporary with location=US
In [7]:
!ls preproc
ls: cannot access 'preproc': No such file or directory

In [8]:
%%bash
gsutil ls gs://${BUCKET}/babyweight/preproc/*-00000*
gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/preproc/eval.csv-00000-of-00012
gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/preproc/train.csv-00000-of-00043

Create TensorFlow model using TensorFlow's Estimator API

First, write an input_fn to read the data.

In [9]:
import shutil
import numpy as np
import tensorflow as tf
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
In [10]:
CSV_COLUMNS = 'weight_pounds,is_male,mother_age,plurality,gestation_weeks,key'.split(',')
LABEL_COLUMN = 'weight_pounds'
KEY_COLUMN = 'key'
DEFAULTS = [[0.0], ['null'], [0.0], ['null'], [0.0], ['nokey']]
TRAIN_STEPS = 1000

def read_dataset(prefix, pattern, batch_size=512):
    # use prefix to create filename
    filename = 'gs://{}/babyweight/preproc/{}*{}*'.format(BUCKET, prefix, pattern)
    if prefix == 'train':
        mode = tf.estimator.ModeKeys.TRAIN
        num_epochs = None # indefinitely
    else:
        mode = tf.estimator.ModeKeys.EVAL
        num_epochs = 1 # end-of-input after this
    
    # the actual input function passed to TensorFlow
    def _input_fn():
        # could be a path to one file or a file pattern.
        input_file_names = tf.train.match_filenames_once(filename)
        filename_queue = tf.train.string_input_producer(
            input_file_names, shuffle=True, num_epochs=num_epochs)
 
        # read CSV
        reader = tf.TextLineReader()
        _, value = reader.read_up_to(filename_queue, num_records=batch_size)
        if mode == tf.estimator.ModeKeys.TRAIN:
          value = tf.train.shuffle_batch([value], batch_size, capacity=10*batch_size, 
                                         min_after_dequeue=batch_size, enqueue_many=True, 
                                         allow_smaller_final_batch=False)
        value_column = tf.expand_dims(value, -1)
        columns = tf.decode_csv(value_column, record_defaults=DEFAULTS)
        features = dict(zip(CSV_COLUMNS, columns))
        features.pop(KEY_COLUMN)
        label = features.pop(LABEL_COLUMN)
        return features, label
  
    return _input_fn

Next, define the feature columns

In [11]:
def get_wide_deep():
    # define column types
    is_male,mother_age,plurality,gestation_weeks = \
        [\
            tf.feature_column.categorical_column_with_vocabulary_list('is_male', 
                        ['True', 'False', 'Unknown']),
            tf.feature_column.numeric_column('mother_age'),
            tf.feature_column.categorical_column_with_vocabulary_list('plurality',
                        ['Single(1)', 'Twins(2)', 'Triplets(3)',
                         'Quadruplets(4)', 'Quintuplets(5)','Multiple(2+)']),
            tf.feature_column.numeric_column('gestation_weeks')
        ]

    # discretize
    age_buckets = tf.feature_column.bucketized_column(mother_age, 
                        boundaries=np.arange(15,45,1).tolist())
    gestation_buckets = tf.feature_column.bucketized_column(gestation_weeks, 
                        boundaries=np.arange(17,47,1).tolist())
      
    # sparse columns are wide 
    wide = [is_male,
            plurality,
            age_buckets,
            gestation_buckets]
    
    # feature cross all the wide columns and embed into a lower dimension
    crossed = tf.feature_column.crossed_column(wide, hash_bucket_size=20000)
    embed = tf.feature_column.embedding_column(crossed, 3)
    
    # continuous columns are deep
    deep = [mother_age,
            gestation_weeks,
            embed]
    return wide, deep

To predict with the TensorFlow model, we also need a serving input function. We will want all the inputs from our user.

In [12]:
def serving_input_fn():
    feature_placeholders = {
        'is_male': tf.placeholder(tf.string, [None]),
        'mother_age': tf.placeholder(tf.float32, [None]),
        'plurality': tf.placeholder(tf.string, [None]),
        'gestation_weeks': tf.placeholder(tf.float32, [None])
    }
    features = {
        key: tf.expand_dims(tensor, -1)
        for key, tensor in feature_placeholders.items()
    }
    return tf.estimator.export.ServingInputReceiver(features, feature_placeholders)

Finally, train!

In [13]:
from tensorflow.contrib.learn.python.learn.utils import saved_model_export_utils
from tensorflow.contrib.learn.python.learn import learn_runner

PATTERN = "00001-of-"  # process only one of the shards, for testing purposes

def train_and_evaluate(output_dir):
    wide, deep = get_wide_deep()
    estimator = tf.estimator.DNNLinearCombinedRegressor(
                         model_dir=output_dir,
                         linear_feature_columns=wide,
                         dnn_feature_columns=deep,
                         dnn_hidden_units=[64, 32])
    train_spec=tf.estimator.TrainSpec(
                         input_fn=read_dataset('train', PATTERN),
                         max_steps=TRAIN_STEPS)
    exporter = tf.estimator.FinalExporter('exporter',serving_input_fn)
    eval_spec=tf.estimator.EvalSpec(
                         input_fn=read_dataset('eval', PATTERN),
                         steps=None,
                         exporters=exporter)
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
    
shutil.rmtree('babyweight_trained', ignore_errors=True) # start fresh each time
train_and_evaluate('babyweight_trained')
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From <ipython-input-10-68490493c118>:22: string_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/input.py:278: input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/input.py:190: limit_epochs (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensors(tensor).repeat(num_epochs)`.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/input.py:199: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/input.py:199: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From <ipython-input-10-68490493c118>:25: TextLineReader.__init__ (from tensorflow.python.ops.io_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.TextLineDataset`.
WARNING:tensorflow:From <ipython-input-10-68490493c118>:30: shuffle_batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.shuffle(min_after_dequeue).batch(batch_size)`.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:3038: CrossedColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:2655: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow_estimator/python/estimator/canned/linear.py:308: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/adagrad.py:76: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py:875: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/input.py:113: RefVariable.count_up_to (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Prefer Dataset.range instead.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py:2322: count_up_to (from tensorflow.python.ops.state_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Prefer Dataset.range instead.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
WARNING:tensorflow:Export includes no default signature!
In [14]:
!ls babyweight_trained
checkpoint
eval
events.out.tfevents.1588665803.tf-tensorboard-1
export
graph.pbtxt
model.ckpt-0.data-00000-of-00002
model.ckpt-0.data-00001-of-00002
model.ckpt-0.index
model.ckpt-0.meta
model.ckpt-1000.data-00000-of-00002
model.ckpt-1000.data-00001-of-00002
model.ckpt-1000.index
model.ckpt-1000.meta

Now that we have the TensorFlow code working on a subset of the data (in the code above, I was reading only the 00001-of-x file), we can package the TensorFlow code up as a Python module and train it on Cloud ML Engine.

Train on Cloud ML Engine

Training on Cloud ML Engine requires:

  1. Making the code a Python package

  2. Using gcloud to submit the training code to Cloud ML Engine </ol>

    The code in model.py is the same as in the above cells. I just moved it to a file so that I could package it up as a module. (explore the directory structure).

In [15]:
%%bash
grep "^def" babyweight/trainer/model.py
def read_dataset(prefix, pattern, batch_size=512):
def get_wide_deep():
def serving_input_fn():
def experiment_fn(output_dir):
def train_and_evaluate(output_dir):

After moving the code to a package, make sure it works standalone. (Note the --pattern and --train_steps lines so that I am not trying to boil the ocean on my laptop). Even then, this takes about a minute in which you won't see any output ...

In [16]:
%%bash
echo "bucket=${BUCKET}"
rm -rf babyweight_trained
export PYTHONPATH=${PYTHONPATH}:${PWD}/babyweight
python -m trainer.task \
   --bucket=${BUCKET} \
   --output_dir=babyweight_trained \
   --job-dir=./tmp \
   --pattern="00001-of-" --train_steps=1000
bucket=qwiklabs-gcp-01-c5358f6c8e9a
WARNING:tensorflow:From /home/jupyter/training-data-analyst/blogs/babyweight/babyweight/trainer/model.py:25: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From /home/jupyter/training-data-analyst/blogs/babyweight/babyweight/trainer/model.py:25: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f631f7eddd0>, '_model_dir': 'babyweight_trained/', '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_experimental_max_worker_delay_secs': None, '_evaluation_master': '', '_eval_distribute': None, '_train_distribute': None, '_master': ''}
INFO:tensorflow:Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/training_util.py:236: initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /home/jupyter/training-data-analyst/blogs/babyweight/babyweight/trainer/model.py:48: The name tf.train.match_filenames_once is deprecated. Please use tf.io.match_filenames_once instead.

WARNING:tensorflow:From /home/jupyter/training-data-analyst/blogs/babyweight/babyweight/trainer/model.py:50: string_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:278: input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:190: limit_epochs (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensors(tensor).repeat(num_epochs)`.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:199: __init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:199: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/jupyter/training-data-analyst/blogs/babyweight/babyweight/trainer/model.py:53: __init__ (from tensorflow.python.ops.io_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.TextLineDataset`.
WARNING:tensorflow:From /home/jupyter/training-data-analyst/blogs/babyweight/babyweight/trainer/model.py:58: shuffle_batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.shuffle(min_after_dequeue).batch(batch_size)`.
WARNING:tensorflow:From /home/jupyter/training-data-analyst/blogs/babyweight/babyweight/trainer/model.py:60: The name tf.decode_csv is deprecated. Please use tf.io.decode_csv instead.

INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling __init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:3038: _num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:2655: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/canned/linear.py:308: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/adagrad.py:76: calling __init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2020-05-05 08:04:21.621394: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  AVX2 FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.

User settings:

   KMP_AFFINITY=granularity=fine,verbose,compact,1,0
   KMP_BLOCKTIME=0
   KMP_DUPLICATE_LIB_OK=True
   KMP_SETTINGS=1
   OMP_NUM_THREADS=4

Effective settings:

   KMP_ABORT_DELAY=0
   KMP_ADAPTIVE_LOCK_PROPS='1,1024'
   KMP_ALIGN_ALLOC=64
   KMP_ALL_THREADPRIVATE=128
   KMP_ATOMIC_MODE=2
   KMP_BLOCKTIME=0
   KMP_CPUINFO_FILE: value is not defined
   KMP_DETERMINISTIC_REDUCTION=false
   KMP_DEVICE_THREAD_LIMIT=2147483647
   KMP_DISP_HAND_THREAD=false
   KMP_DISP_NUM_BUFFERS=7
   KMP_DUPLICATE_LIB_OK=true
   KMP_FORCE_REDUCTION: value is not defined
   KMP_FOREIGN_THREADS_THREADPRIVATE=true
   KMP_FORKJOIN_BARRIER='2,2'
   KMP_FORKJOIN_BARRIER_PATTERN='hyper,hyper'
   KMP_FORKJOIN_FRAMES=true
   KMP_FORKJOIN_FRAMES_MODE=3
   KMP_GTID_MODE=3
   KMP_HANDLE_SIGNALS=false
   KMP_HOT_TEAMS_MAX_LEVEL=1
   KMP_HOT_TEAMS_MODE=0
   KMP_INIT_AT_FORK=true
   KMP_INIT_WAIT=2048
   KMP_ITT_PREPARE_DELAY=0
   KMP_LIBRARY=throughput
   KMP_LOCK_KIND=queuing
   KMP_MALLOC_POOL_INCR=1M
   KMP_NEXT_WAIT=1024
   KMP_NUM_LOCKS_IN_BLOCK=1
   KMP_PLAIN_BARRIER='2,2'
   KMP_PLAIN_BARRIER_PATTERN='hyper,hyper'
   KMP_REDUCTION_BARRIER='1,1'
   KMP_REDUCTION_BARRIER_PATTERN='hyper,hyper'
   KMP_SCHEDULE='static,balanced;guided,iterative'
   KMP_SETTINGS=true
   KMP_SPIN_BACKOFF_PARAMS='4096,100'
   KMP_STACKOFFSET=64
   KMP_STACKPAD=0
   KMP_STACKSIZE=4M
   KMP_STORAGE_MAP=false
   KMP_TASKING=2
   KMP_TASKLOOP_MIN_TASKS=0
   KMP_TASK_STEALING_CONSTRAINT=1
   KMP_TEAMS_THREAD_LIMIT=4
   KMP_TOPOLOGY_METHOD=all
   KMP_USER_LEVEL_MWAIT=false
   KMP_VERSION=false
   KMP_WARNINGS=true
   OMP_AFFINITY_FORMAT='OMP: pid %P tid %T thread %n bound to OS proc set {%a}'
   OMP_ALLOCATOR=omp_default_mem_alloc
   OMP_CANCELLATION=false
   OMP_DEFAULT_DEVICE=0
   OMP_DISPLAY_AFFINITY=false
   OMP_DISPLAY_ENV=false
   OMP_DYNAMIC=false
   OMP_MAX_ACTIVE_LEVELS=2147483647
   OMP_MAX_TASK_PRIORITY=0
   OMP_NESTED=false
   OMP_NUM_THREADS='4'
   OMP_PLACES: value is not defined
   OMP_PROC_BIND='intel'
   OMP_SCHEDULE='static'
   OMP_STACKSIZE=4M
   OMP_TARGET_OFFLOAD=DEFAULT
   OMP_THREAD_LIMIT=2147483647
   OMP_TOOL=enabled
   OMP_TOOL_LIBRARIES: value is not defined
   OMP_WAIT_POLICY=PASSIVE
   KMP_AFFINITY='verbose,warnings,respect,granularity=fine,compact,1,0'

2020-05-05 08:04:21.665589: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2020-05-05 08:04:21.666293: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5615844ef330 executing computations on platform Host. Devices:
2020-05-05 08:04:21.666345: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-3
OMP: Info #156: KMP_AFFINITY: 4 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 2 cores/pkg x 2 threads/core (2 total cores)
OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 0 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6298 thread 0 bound to OS proc set 0
2020-05-05 08:04:21.668137: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2020-05-05 08:04:21.824039: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py:875: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6322 thread 1 bound to OS proc set 1
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6324 thread 2 bound to OS proc set 2
INFO:tensorflow:Saving checkpoints for 0 into babyweight_trained/model.ckpt.
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6320 thread 3 bound to OS proc set 3
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6328 thread 5 bound to OS proc set 1
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6327 thread 4 bound to OS proc set 0
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6329 thread 6 bound to OS proc set 2
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6321 thread 7 bound to OS proc set 3
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6330 thread 8 bound to OS proc set 0
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6331 thread 9 bound to OS proc set 1
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6332 thread 10 bound to OS proc set 2
INFO:tensorflow:loss = 1346.5928, step = 1
INFO:tensorflow:global_step/sec: 64.0357
INFO:tensorflow:loss = 568.1858, step = 101 (1.562 sec)
INFO:tensorflow:global_step/sec: 114.582
INFO:tensorflow:loss = 606.4372, step = 201 (0.873 sec)
INFO:tensorflow:global_step/sec: 118.277
INFO:tensorflow:loss = 602.66724, step = 301 (0.845 sec)
INFO:tensorflow:global_step/sec: 114.308
INFO:tensorflow:loss = 565.3104, step = 401 (0.875 sec)
INFO:tensorflow:global_step/sec: 118.335
INFO:tensorflow:loss = 612.0133, step = 501 (0.845 sec)
INFO:tensorflow:global_step/sec: 115.917
INFO:tensorflow:loss = 547.14056, step = 601 (0.863 sec)
INFO:tensorflow:global_step/sec: 114.51
INFO:tensorflow:loss = 603.563, step = 701 (0.874 sec)
INFO:tensorflow:global_step/sec: 113.587
INFO:tensorflow:loss = 571.4082, step = 801 (0.880 sec)
INFO:tensorflow:global_step/sec: 115.43
INFO:tensorflow:loss = 503.78738, step = 901 (0.867 sec)
INFO:tensorflow:Saving checkpoints for 1000 into babyweight_trained/model.ckpt.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:113: count_up_to (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Prefer Dataset.range instead.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py:2322: count_up_to (from tensorflow.python.ops.state_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Prefer Dataset.range instead.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-05-05T08:04:34Z
INFO:tensorflow:Graph was finalized.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from babyweight_trained/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6367 thread 11 bound to OS proc set 3
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6368 thread 11 bound to OS proc set 3
INFO:tensorflow:Finished evaluation at 2020-05-05-08:04:38
INFO:tensorflow:Saving dict for global step 1000: average_loss = 1.5951903, global_step = 1000, label/mean = 7.329146, loss = 815.47687, prediction/mean = 6.7010813
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: babyweight_trained/model.ckpt-1000
WARNING:tensorflow:From /home/jupyter/training-data-analyst/blogs/babyweight/babyweight/trainer/model.py:105: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:
INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'plurality': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=string>, 'mother_age': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>, 'gestation_weeks': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'is_male': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=string>}
INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'plurality': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=string>, 'mother_age': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>, 'gestation_weeks': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'is_male': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=string>}
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from babyweight_trained/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: babyweight_trained/export/exporter/temp-1588665878/saved_model.pb
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6323 thread 11 bound to OS proc set 3
OMP: Info #250: KMP_AFFINITY: pid 6298 tid 6325 thread 1 bound to OS proc set 1
INFO:tensorflow:Loss for final step: 528.78296.

Once the code works in standalone mode, you can run it on Cloud ML Engine. Note that doing it on the entire dataset will take a while. The training run took about an hour for me on STANDARD_1. So, for the purposes of Qwiklabs, this is done on a partial dataset on a smaller tier (BASIC_1). Monitor the progress of this job in the Cloud ML Engine section of the GCP Console. Once you see a green check mark, run the next Datalab cell.

In [17]:
%%bash
OUTDIR=gs://${BUCKET}/babyweight/trained_model
JOBNAME=babyweight_$(date -u +%y%m%d_%H%M%S)
echo $OUTDIR $REGION $JOBNAME
gsutil -m rm -rf $OUTDIR
gcloud ai-platform jobs submit training $JOBNAME \
   --region=$REGION \
   --module-name=trainer.task \
   --package-path=$(pwd)/babyweight/trainer \
   --job-dir=$OUTDIR \
   --staging-bucket=gs://$BUCKET \
   --scale-tier=BASIC \
   --runtime-version 1.14 \
   -- \
   --bucket=${BUCKET} \
   --output_dir=${OUTDIR} \
   --pattern="00001-of-" \
   --train_steps=2000
gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model us-central1 babyweight_200505_080457
jobId: babyweight_200505_080457
state: QUEUED
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/checkpoint#1588665743275030...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/eval/events.out.tfevents.1529348264.cmle-training-master-a137ac0fff-0-9q8r4#1588665743242178...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/events.out.tfevents.1529347276.cmle-training-master-a137ac0fff-0-9q8r4#1588665743260230...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/export/exporter/1529355466/saved_model.pb#1588665743280095...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/export/exporter/1529355466/variables/variables.data-00000-of-00001#1588665743352354...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/graph.pbtxt#1588665743349579...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-342784.data-00001-of-00003#1588665743375799...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-342784.data-00002-of-00003#1588665743383994...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-342784.meta#1588665743399610...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/export/exporter/1529355466/variables/variables.index#1588665743373348...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-342784.data-00000-of-00003#1588665743429000...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-376661.data-00002-of-00003#1588665743447344...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-390628.data-00000-of-00003#1588665743483969...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-376661.data-00001-of-00003#1588665743469962...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-390628.data-00001-of-00003#1588665743479096...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-342784.index#1588665743363100...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-376661.data-00000-of-00003#1588665743395139...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-376661.meta#1588665743442538...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-376661.index#1588665743447388...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-390628.data-00002-of-00003#1588665743532640...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-390628.index#1588665743553147...
Removing gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/model.ckpt-390628.meta#1588665743562576...
/ [22/22 objects] 100% Done                                                     
Operation completed over 22 objects.                                             
Job [babyweight_200505_080457] submitted successfully.
Your job is still active. You may view the status of your job with the command

  $ gcloud ai-platform jobs describe babyweight_200505_080457

or continue streaming the logs with the command

  $ gcloud ai-platform jobs stream-logs babyweight_200505_080457

Training finished with a RMSE of 1 lb. Obviously, this is our first model. We could probably add in some features, discretize the mother's age, and do some hyper-parameter tuning to get to a lower RMSE. I'll leave that to you. If you create a better model, I'd love to hear about it -- please do write a short blog post about what you did, and tweet it at me -- @lak_gcp.

Monitor training with TensorBoard

To activate TensorBoard within the JupyterLab UI navigate to "File" - "New Launcher". Then double-click the 'Tensorboard' icon on the bottom row.

TensorBoard 1 will appear in the new tab. Navigate through the three tabs to see the active TensorBoard. The 'Graphs' and 'Projector' tabs offer very interesting information including the ability to replay the tests.

You may close the TensorBoard tab when you are finished exploring.

Make sure that training is complete before you move to the next step. Visit the Cloud ML Engine web console and wait until you see a green check mark before you proceed

Deploy trained model

Deploying the trained model to act as a REST web service is a simple gcloud call.

In [19]:
%%bash
gsutil ls gs://${BUCKET}/babyweight/trained_model/export/exporter
gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/export/exporter/
gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/export/exporter/1588666018/

If you are getting error then wait for a minute and run the command again.

In [20]:
%%bash
MODEL_NAME="babyweight"
MODEL_VERSION="qwiklab"
MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/babyweight/trained_model/export/exporter/ | tail -1)
echo "Deleting and deploying $MODEL_NAME $MODEL_VERSION from $MODEL_LOCATION ... this will take a few minutes"
#gcloud ai-platform versions delete ${MODEL_VERSION} --model ${MODEL_NAME}
#gcloud ai-platform models delete ${MODEL_NAME}
gcloud ai-platform models create ${MODEL_NAME} --regions $REGION
gcloud ai-platform versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version 1.14
Deleting and deploying babyweight qwiklab from gs://qwiklabs-gcp-01-c5358f6c8e9a/babyweight/trained_model/export/exporter/1588666018/ ... this will take a few minutes
Created ml engine model [projects/qwiklabs-gcp-01-c5358f6c8e9a/models/babyweight].
Creating version (this might take a few minutes)......
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done.

Use model to predict

Send a JSON request to the endpoint of the service to make it predict a baby's weight ... I am going to try out how well the model would have predicted the weights of our two kids and a couple of variations while we are at it ...

In [21]:
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
import json

credentials = GoogleCredentials.get_application_default()
api = discovery.build('ml', 'v1', credentials=credentials)

request_data = {'instances':
  [
    {
      'is_male': 'True',
      'mother_age': 26.0,
      'plurality': 'Single(1)',
      'gestation_weeks': 39
    },
    {
      'is_male': 'False',
      'mother_age': 29.0,
      'plurality': 'Single(1)',
      'gestation_weeks': 38
    },
    {
      'is_male': 'True',
      'mother_age': 26.0,
      'plurality': 'Triplets(3)',
      'gestation_weeks': 39
    },
    {
      'is_male': 'Unknown',
      'mother_age': 29.0,
      'plurality': 'Multiple(2+)',
      'gestation_weeks': 38
    },
  ]
}

parent = 'projects/%s/models/%s/versions/%s' % (PROJECT, 'babyweight', 'qwiklab')
response = api.projects().predict(body=request_data, name=parent).execute()
print ("response={0}".format(response))
response={'predictions': [{'predictions': [7.026079177856445]}, {'predictions': [7.284391403198242]}, {'predictions': [6.670483112335205]}, {'predictions': [6.962189674377441]}]}

Copyright 2018 Google Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License