The needs for orchestrating workloads for industrial analytics go above and beyond the standard ML pipelines. Several cases involve a combination of domain specific models such as physics based models, probabilistic models and AI/ML models. In many cases, each of these models could be written in a different programming language and/or with several different libraries within the same programming language. This necessitates the need for containerized and orchestrated workloads. In some cases, different parts of the pipeline could be owned by different groups that may not want to expose the IP.
In addition, model calibration and optimization use cases require more than a simple DAG orchestration. In many of these cases, cyclic and conditional workflows are needed. When dealing with Industrial IoT applications, hundreds if not thousands of models are normally built to represent a single asset. In addition, optimization and calibration use cases require executing the pipelines thousands of times for each optimization or calibration exercise. Hence scaling such complex pipelines is a key component of industrial analytics systems. We will showcase building and managing such complex pipelines using various GCP services such as GKE, TFX, PubSub and BigTable.
Build with Google Cloud → https://bit.ly/2WQC04q
Next ’19 Data Analytics Sessions here → https://bit.ly/Next19DataAnalytics
Next ‘19 All Sessions playlist → https://bit.ly/Next19AllSessions
Subscribe to the GCP Channel → https://bit.ly/GCloudPlatform
Speaker(s): Alexandre Iankoulski, Fabio Nonato
Session ID: DA226
product:BigQuery,Cloud Storage,Cloud Pub/Sub;