kflansburg.com
Discussions on Rust, Kubernetes, architecture, and more.

Ordered Stateful Streaming | Failure Modes

-- 841 Words
Spark, Streaming, Scala,

In the fourth post of this series I will outline several sources of failure in streaming applications. In each case I discuss the potential for data loss and possible mitigations. This analysis can drive decisions on the tradeoff between infrastructure costs and the upper bound on data loss.

Read more...

Ordered Stateful Streaming | Operating Pipelines

-- 2221 Words
Spark, Streaming, Scala,

In the third post of this series I will share operational and practical issues that I have encountered when developing and hosting long-running Spark streaming applications. This includes tips that will be useful to those who are first starting out with Scala and Spark, as well as insights on performance and reliability that will be useful to those who are more experienced.

Read more...

Krator: My God, it's Full of States!

-- 56 Words
Rust, Kubernetes, Krustlet,
At the beginning of February, I published my second guest post on Deis Labs' blog, which introduced Krator. Pronounced “crater”, this crate allows you to build Kubernetes Operators in Rust, using the state machine API that we designed for Krustlet last year. Check out the post for an example of writing an Operator with this crate!

Ordered Stateful Streaming | Selecting a Framework

-- 2011 Words
Spark, Streaming, Scala,

In the second post of this series I explore the strengths and weaknesses of several popular streaming frameworks. This analysis was performed a couple of years ago with a particular application in mind. These frameworks have since improved, but this post should provide some insight into the tradeoffs and decisions involved when designing streaming applications, and lessons can be learned from choices that did and did not pay off.

Read more...

Ordered Stateful Streaming | Introduction

-- 1792 Words
Spark, Streaming, Scala,

I recently spent two weeks tracking down a subtle bug in a Spark Structured Streaming application which I have been maintaining for several years. Having dealt with many such time-consuming bugs over the years, I’ve decided to compile my experiences working with ordered, stateful streaming applications into a series of posts. This series will serve as an introductory guide to the design and operation of stateful streaming pipelines, and hopefully spur some further development to simplify this process in the future.

Read more...

Rust Continuous Delivery

-- 2276 Words
AWS, Terraform, Rust, CICD, Kubernetes,

Over the last few years I have iterated several times on continuous delivery pipelines for Rust applications. Designing these pipelines involves balancing a number of factors including cost, complexity, ergonomics, and rigor. In this post I will describe several of these iterations, lessons learned, and share my most recent solution in detail.

Read more...

A Fistful of States: More State Machine Patterns in Rust

-- 37 Words
Rust, Kubernetes, Krustlet,
My recent guest post on Deis Labs' blog offers a deep dive into my work on Krustlet, including the design and implementation of a flexible state machine API for specifying custom Kubelet behavior in a type-safe framework.
1 of 1