Building a cloud-scale central retry service for Data Pipelines.

Current Data Pipelines are built using various queueing and messaging patterns. For the purpose of this blog post, I’m going to break them down into three categories by borrowing terms from ZeroMQ and others. The first two, Publish-Subscribe and Push-Pull are in-memory (volatile memory). The third, we’ll call Streaming which is persisted on disk (non-volatile memory).

Lately, I have found myself spending a decent amount of time on open-source data projects. With that, I hope to keep this blog up to date on what I am working on in regartds to Golang and open-source data projects.

Setting up GraphQL Ruby with Pundit