Press "Enter" to skip to content

Analytics Orchestration at Scale with Kubernetes, Tensorflow, and Kubeflow (Cloud Next '19)

The needs for orchestrating workloads for industrial analytics go above and beyond the standard ML pipelines. Several cases involve a combination of domain specific models such as physics based models, probabilistic models and AI/ML models. In many cases, each of these models could be written in a different programming language and/or with several different libraries within the same programming language. This necessitates the need for containerized and orchestrated workloads. In some cases, different parts of the pipeline could be owned by different groups that may not want to expose the IP.
In addition, model calibration and optimization use cases require more than a simple DAG orchestration. In many of these cases, cyclic and conditional workflows are needed. When dealing with Industrial IoT applications, hundreds if not thousands of models are normally built to represent a single asset. In addition, optimization and calibration use cases require executing the pipelines thousands of times for each optimization or calibration exercise. Hence scaling such complex pipelines is a key component of industrial analytics systems. We will showcase building and managing such complex pipelines using various GCP services such as GKE, TFX, PubSub and BigTable.

Build with Google Cloud →

Watch more:
Next ’19 Data Analytics Sessions here →
Next ‘19 All Sessions playlist →

Subscribe to the GCP Channel →

Speaker(s): Alexandre Iankoulski, Fabio Nonato

Session ID: DA226
product:BigQuery,Cloud Storage,Cloud Pub/Sub;

0 0 vote
Article Rating
Notify of
Inline Feedbacks
View all comments
Would love your thoughts, please comment.x