avatar

A blog on building teams, data pipelines, and AI.

Data Pipelines: Part 1 - Queuing and Messaging Patterns

Current Data Pipelines are built using various queueing and messaging patterns. For the purpose of this blog post, I’m going to break them down into three categories by borrowing terms from ZeroMQ and others. The first two, Publish-Subscribe and Push-Pull are in-memory (volatile memory). The third, we’ll call Streaming which is persisted on disk (non-volatile memory).

Forward

Lately, I have found myself spending a decent amount of time on open-source data projects. With that, I hope to keep this blog up to date on what I am working on in regartds to Golang and open-source data projects.