State vs. Flow Data Architecture in the Financial Sector | Whiteboard Walkthrough

Contributed by

17 min read

In this Whiteboard Walkthrough, MapR’s Chief Application Architect, Ted Dunning, explains the move from state to flow and shows how it works in a financial services example. Ted describes the revolution underway in moving from a traditional system with multiple programs built around a shared database to a new flow-based system that instead uses a shared state queue in the form of a message stream built with technology such as Apache Kafka or MapR Event Store. This new architecture lets decisions be made locally and supports a micro-services style approach.

For more, you may find these resources useful:

Here's the transcription:

Hi, I'm Ted Dunning, I'm chief application architect at MapR and what I'd like to talk about today is a bit of a revolution in how we're building large scale systems. How our customers are building large scale systems. The issue is that we're moving from a world of state and applications built around the abstraction of a shared state into a world characterize by flow of information, where there is no shared single state.

Now, building systems in that new style can be tricky and in fact you can wind up with some very difficult complications. But if you do it right, you can wind up with some very high performance, very large scale systems that can span the globe, that still work very, very predictably.

This gives punitively simpler semantics, because most databases provide the ability to make multiple changes exactly at the same time. Exactly, in the sense that either nobody sees any of the changes, or they see all of the changes. Now not all observers may agree about time and when the changes appear, but that brings in the issue of time, which is difficult to deal with in computing anyway. Soon after the change everybody will see the entire change and shortly before the change nobody will see it. This simpler semantics is organized around this idea of a transaction. A database transaction.

This gives punitively simpler semantics, because most databases provide the ability to make multiple changes exactly at the same time. Exactly, in the sense that either nobody sees any of the changes, or they see all of the changes. Now not all observers may agree about time and when the changes appear, but that brings in the issue of time, which is difficult to deal with in computing anyway. Soon after the change everybody will see the entire change and shortly before the change nobody will see it. This simpler semantics is organized around this idea of a transaction. A database transaction.

The "ish" on the simpler semantics comes from the fact that once you share state you have a complicated semantics, just inherent like. Making sure that this hand changes something that this hand wasn't expecting to be changed requires that you use transactions very carefully. It's very common for shared databases to have subtle errors in them as a result. This is also a familiar idiom. People are used to dealing with the complications of shared state. The fundamentals underneath it are control-flow, normal programming constructs that we have. Perhaps adjusted somewhat in a database world, where certain control flow elements statements that are executed are actually database transaction things, but the basic idea is we have loops and things like that, and state updates, global state updates. Those are the primitive, the fundamental operations in a state based world.

In a flow based world things are a bit different. We have no sharing. That doesn't mean we have no databases, it means we have no globally shared, globally accessible databases. Services, and this is kind of a micro-service sort of world that we're talking about there, services can have their own databases, but the contract is that they don't share them. They will share flows, either in or out, but they will maintain their own database privately, and therefore subject to change without committee meetings. We need good idioms, because once we go to flow we expose all of the complexities of having concurrent systems. Concurrent systems are difficult to design, but if we have certain kinds of idioms, such as data always flows without cycles through some large portion of the system, we can mitigate those difficulties, mitigate those complexities, and have easy to write systems. This is somewhat the equivalent of using transactions in the state world, which was an idiom that we would use, a primitive capability that we would use, to mitigate the complexities of concurrent systems.

In many cases flow is what we wanted all along. Flow is what we meant all these years when we would draw system diagrams that involved blocks with arrows going to other blocks. Typically when we built that kind of system with blocks and arrows, we kind of ad hoc invented the arrows, or we would use a database to store the data in the arrows. We had real troubles when we wanted to rerun the system or something like that. In many ways, flow is what we needed, what we wanted to use all along. I know that I've built many systems in the past where if I had good flow semantics, good flow primitive elements, I could have built a much better system. The fundamentals of a flow program are control flow, the programs work basically the same way, each component of it, but now we don't have shared state. What we do is we send messages to cues, which are often named by topics and we record off-sets in these message cues, in these topics. The primitive elements are very different. One of them is the same, one of them is completely different.

Let me give you just one example, this is a very short thing, I can't go in to all of the cool things we can do with this, but take the canonical textbook case that people talk about as a motivating need for transactions, and that is bank accounts. They always talk about, I want to debit one account and credit the other account at exactly the same time. I want to take $5 from Bob and give it to Carol and always have the sum of all of the accounts have exactly the same value. The $5 may be on Bob's side, it may be on Carol's side, but it's never on both, and it's never on neither. That's the statement, that's the idea of doing it in a database, in a flow world, and in fact in the real world, we know it doesn't work that way. I write the check, or Bob writes the check, gives it to Carol, Carol puts it in her bank and there's a hold placed on the funds. They fictionally say, oh yeah, sure you've got the $5, but you can't spend it yet. Their bank sends a copy of the check, electronically usually now to my bank, my bank says oh that's cool, and subtract some money out of my account, before they'd subtract it out of Carol's account. Before they add it to Carol's account.

The things don't happen at the same time. The sum of the accounts is not always constant. That's the real world. That's how it works. In a message world, we can actually maintain the invariant, but still have the distributed nature. The idea is that instead of decrementing one account and incrementing another account putatively inside a transaction, what we do is send a high level business event. The high level business event is $5, goes from Bob to Carol. There would be more data in it than that, this is just a tiny example. That's a message in shared state cue. That might be one cue in one place, one Kafka topic, say in one Kafka cluster, or it might be echoes of the same topic in many places. If you're using MapR, you could use MapR Event Store, you could replicate the stream to many, many data centers, they could all have the same topic in them, and once that message is put into that topic it appears everywhere with exactly the same off-set. It may not appear in all of the copies right away due to network limitations, but as soon as the network partition heals, if any, it will appear everywhere, and everybody will have a consistent view of that as long as the topic has the insertion from a single place at a single time.

We get that business event and Bank 1, this is Bob's bank, will be trolling through all of these messages for a message that refers to one of their account holders, say Bob. They see the transaction for Bob, and so they're down here at this time, and so they subtract $5 from Bob's account. They do this without reference to the other bank, Carol's bank. Carol's bank on the other side, before they've seen that message in the event stream, does not add the $5, after they've seen that message in the event stream possibly after a buffer time, after the event is officially occurred they will add $5 to Carol's account.

Being able to build business events like that and make this into a flow idiom depends critically on the fact that we can make this be an unconditional transaction. It will take place. All the things will make its effect known regardless. With bank accounts that isn't quite true. Bob may not have $5, he may only have $2. If so, what will happen is bank 1, Bob's bank, when they see this transaction will emit another business event saying reversal of previous transaction due to insufficient funds, so that Carol's bank, who is of course processing these a little bit delayed all of the deposits a little delayed, banks will tend to do that, they'll process the withdrawals immediately and the deposits after a bit of a delay. They will see two transactions within that grace period. One is the forward transaction, $5 from Bob to Carol, and then they will see the reversal $5 Carol to Bob insufficient funds reversal of account. They will have no net effect on Carol's account, and likewise Bob's bank will have no effect on their account other than to flag that there was an insufficient funds transfer.

That's how you make, very often these sorts of transactions immutable and unconditional. This transaction, the $5 from Bob to Carol is unconditional if it fails, or if it could not be applied by any of the parties, we must insert reversal transactions into the stream that will be processed. That's kind of like it is in the real world. You can't always coordinate between distributed entities. The message stream allows you to have some coordination and some agreeable ground truths.

The flow diagram for this is that we put the message in to this transfer cue, this topic, which is a message cue that's why we draw it as a sideways trashcan instead of a vertical trashcan like a database, and both banks are listening to the same topic. They will both get the same messages, and they can agree on the off-sets of all of the messages so they can agree that they have processed all of these transactions up to a certain point, or they have not. They can build their own little databases on their own side of all of the state of all of the accounts that they are handling. That database in some sense you can view as the reification, as the instantiation, as the flow, the evanescent flow made solid. The database will contain the state of all the effect of all of the business level events that have been processed so far in concrete form, but it's not a global database. The database in bank 1 and bank 2 will not be the same, because they will not have processed all the same messages yet.

If you want to go crazy about this, I mean it's really not that complicated, you just make sure that these events happened. If you want to go kind of crazy about it, and talk about category theory, the business events must be monads. That's the idea of a function that applies to state. Each of these guys can have their own state variable that has all these functions applied to it one after another always in the same order, and they can guarantee that at off-set 35,000 they would have the same value of the state at that point. Now they may have gone on from that or may not have reached that yet so the actual state that they have right now would be different, but they agree that if they were to stop at that transaction they would have exactly the same state.

This is the fundamental way that we convert programs from state-based to flow-based. We change from using a shared database to using a shared flow. The flow itself contains high-level events, not table-level events, and we're free then in all the distributed entities to apply those events however seems appropriate within our context. For bank 1 that means debiting Bob's account, but they have no account for Carol so they ignore that aspect of it. Bank 2 increments Carol's account because they have Carol's account, but they have no account for Bob, and so they don't do anything with that. They both interpret the message as appropriate. They have a flow program now and they can start decomposing the system into microservices, which then improves the entire function of the organization, because it means decisions can be made locally. That's the essence. It really isn't that hard a concept to implement, except that history may have given you a state-based world that you have to rethink in terms of these business events. I look forward to hearing from you, but this is really an exciting change in how we can build these large-scale systems. Finally, we have arrows and that's so cool. Thanks very much!

This blog post was published October 05, 2016.

50,000+ of the smartest have already joined!

Stay ahead of the bleeding edge...get the best of Big Data in your inbox.

Get our latest posts in your inbox

Subscribe Now