Qunfei@Data, ML and AI Architecture

With a high-frequency trading requirement, real-time and low-latency becomes the critical need in the trading system.

I was a software architect who involved to upgrade a legacy system into a real-time gambling trading system for a billion-sized business. Based on the maturity of using streams, new machine learning ideas were down to earth significantly. Today I want to share this five-year painful but worthy journey from Zero to Master on Kafka stream with python in this series. Besides the high-performance, the developing tips consolidate with stream system design, DevOps operation and quality assurance to build an enterprise python application.

There are eight sections of this series. It covers the concepts, patterns and architectures with five hands-on examples,

• Get Started Stream Processing
• Kafka Python Library Architecture
• Stream Process Design Pattern
• Stream Processing Operation
• Building an Enterprise Application
• Stream Application Test
• Monitoring Stream Application
• Stream System Capacity Design

Furthermore, to avoid the gap from concepts into practice. All five examples with machine learning libraries inspire all data scientists and engineers to transform their model and service into event-driven architecture faster and easier in action. In the meanwhile for business, to provide a new landscape to extend the uncovered business model with stream technology.  In this case, does business drive technology or technology drive business?

Like the Concorde airline, the business needed faster airline even in 1969, but there was no mature-ready technology to run at that moment.

• 

So I applied real-time stream technology into exciting topics like an earthquake, stock market, chatbot, smart-price with example as following,

Example	Stream Design Pattern	Dependency	Keywords	Description
HelloWorld	Single-Event Processing	Stream	Producer, Consumer	Simple event processing
WordCount	Processing with local State	Stream, Table, Web service	State, expose state, web service	Simple event processing with state
Earthquake	Processing with external lookup	Mysql, HttpClient, Cron jobs, Timers, de/serialization/, schema register	External db, api, partitions, gisui, message serialisation/deserilaztion	Simple earthquake data in realtime GIS-UI
StockMarket	Multiphase processing/repartitioning	Stream, Table, Batch-Training, Real-time recommendation	External api, batch-training, real-time recommendation, logs, monitoring	Simple stock market recommendation system
FraudDetection	Multiphase processing	Stream, Table, Juypter notebook, autoML	Windows, Real-time prediction, metrics, autoML	Simple fraud detection in insurance claim
Chatbot	Multiphase processing	Real-time Web & Websocket, NLP rasa, spacy	NLP, Real-time response	Simple NLP process and response in real-time
SmartPrice	Multiphase processing	Incremental learning model,	Feature Store, Incremental learning model	Simple incremental learning model

In the end, this series is helping you to build confidence in adopting the steam-system, to choose a different stream design pattern, to avoid some apparent traps, to design your first stream production-ready and enterprise-ready application in python. Let’s shift stream technology into new business models.