“Near Real-time Data Pipelines,” or NRDP, the newest and hottest trend in Data Engineering… But what is it really?
- It’s not Hadoop batch processing MapReduce.
- It’s not Airflow, though it could be if you hired a large team.
- It’s not a way of modelling ultra-micro-services, though it could be
- It’s not a tight vertical model with class->serializer-endpoint on mvc
What is a NRDP? It is Augmented Intelligence Process Introspection or AIPI, which is applied data science and AI which is injected into a live transaction stream.
NRDP can be placed mid-stream, upstream or downstream in any existing process and can fork your process decisions “in flight” so to speak.
Who has this? Nobody. Well, some advanced systems are using this, but it takes a large team of Data Engineers to accomplish this. It’s a race.
Crowd Inc has demonstrable evidence of creating NRDP environments on Kubernetes/Docker on EKS/ECS Pods micro-services and class model architectures.
Why does it matter that these are discrete and “guest services?”
- Because they are self-contained functions (Think Lambda/Step)
- They are independent functions
- They do not alter the process, they alter the pathway
- They are completely uninterested in anything but their own function
Call me at 310-294-0813 if you are interested in learning more