Streaming Architecture:

New Designs Using Apache Kafka and MapR Streams

by Ted Dunning & Ellen Friedman

Fraud Detection with Streaming Data

Rapid detection of credit card fraud is just one of many examples from the financial sector where streaming data is an important part of the solution. In this chapter, we’ll show how stream-based architecture can provide a better architectural foundation for credit card fraud detection and, if designed in a clever way, also provide benefits for projects beyond the first specific goal for fraud protection.

Thinking of the example in this way, we have two major goals in our design:

  • Goal 1

    When a customer uses a credit card to do a transaction, the vendor needs a fast response to the question, “Is it fraud?”

  • Goal 2

    We need to keep a history of fraud decisions the system has made. That history of decisions should be available to other applications and services within the organization as well as updating the database within the fraud system.

Card Velocity

For our credit card example, in order to highlight the architectural aspects, we take a highly simplified view of how to actually detect fraud. Specifically, we’ll look at a property known as card velocity and use it as an indicator of the likelihood of fraudulent activity. The idea behind card velocity is quite straightforward. Suppose you see that a credit card is used at a point of sale (POS) in London, and three minutes later, the same card is used at a POS in Sydney, Australia. What do you conclude? Until someone invents an effective molecular transporter or time machine, it’s likely that one or both credit card transactions are fraudulent.

Identifying fraudulent activity sounds easy to do, and basically it is, except that these decisions have to be made for millions of transactions, and very quickly. After a decision is made about the relative likelihood of fraud for each one, the result must be returned reliably to the POS with a latency of just a few dozen milliseconds. That’s where machine-based decisions and a system for handling message streams become important. Figure 6-1 shows the high-level structure of our sample system. In the figure, the decision engine used to decide whether or not a transaction appears to be fraudulent is represented by a question mark in a box; we will explain what lies within the box next. The point is that many POS terminals can send requests for decisions to the same decision engine.

ndsa 0601
Figure 6-1. When a credit card transaction takes place, a request is sent from the POS to a data center to ask if the transaction is likely to be fraudulent (yes/no?). The request identifies the event with data that includes location and time of the transaction and the card number. This simple diagram highlights the idea that a huge number of credit card transaction requests are coming in at nearly the same time. The decision needs to be made and the answer provided within about 50–100 milliseconds.

Keep in mind that our example is simplified for sake of illustration. Card velocity is, in fact, often used in real fraud detection systems as one indicator of credit card fraud, but it’s only one of hundreds of such indicators. These systems tend to be complex and involve weighing combinations of many indicators. We are focusing on the overall architecture, however, so we will keep this example simple.

Fast Response Decision to the Question: “Is It Fraud?”

We will use card velocity as the method to detect an anomalous transaction that might signal a fraud attempt. To do that, each transaction needs to include information about the purchase location and time, as well as some way to identify it as being associated with a particular credit card, such as the card number. This information needs to be conveyed with the request to the fraud detector application at a data center for each request, as shown in Figure 6-2.

We won’t go into the details of the fraud detector application here (we have discussed similar cases in other publications), but at a high level, what is involved is a system trained to recognize what represents a range for normal behavior related to the time interval between two successive transactions and the distance between locations. In our simplified example, the model only needs to retain the previous location. For building a more complex model, however, we would need to collect card transaction histories and use this data to train a more sophisticated detection model. In anticipation of that, we design the architecture such that collecting transaction histories is easy, even though we don’t necessarily need to do it for the current step.

To actually make a fraud decision, we need to look up the previous use of the card and subtract that location and time of use from the data for the current transaction to compute the card velocity. We use this computation to decide if the velocity is too high to be trusted.

These steps are depicted in the architecture diagram shown in Figure 6-2.

ndsa 0602
Figure 6-2. The diagram shows an architectural design for the data flow needed to meet the goal of a fast response for credit card transaction verification. The fraud detector makes a decision about card activity and returns that decision back to the POS. It also records the transaction information and decision so that an updater program can make a record of the last use. The update provides information about the last card use, which in turn will be used as reference data when the next fraud decision is made.

Notice that the POS sends a query to the fraud detector and expects a response to be returned directly. We don’t implement this step using streaming because a query-response style is a more appropriate match for user expectations in this situation. However, we do publish the output of the detector, including the card and transaction information and the fraud decision, to a message stream for later processing. This stream is shown as a horizontal tube labeled “card activity” in Figure 6-2.

Publishing historical transactions to a stream is appropriate because we don’t want the fraud detector to know or care how the historical transactions are used—we want to completely decouple any current or future consumers of that data from the fraud detector itself. Having a stream of transaction information available actually helps build the more complex fraud detection mechanisms that would be used in reality.

On the other hand, even in our simplified card velocity example, a database is needed to store the last transaction information so that it is easy to look up the transaction by card number. It is tempting to build this system by having the fraud detector directly update the database each time a decision is made. In practice, it is actually better to send all transactions to a message stream that is outside the fraud detector and then reimport that information into the database inside the fraud detector. This design is nonintuitive, but the rationale becomes clearer as we talk about scaling our fraud detector prototype. The database, by the way, might be built using Apache HBase or MapR’s integrated NoSQL document-style database, MapR JSON DB.

The data flow shown in Figure 6-1 meets the needs of Goal 1, but by using the message stream (Kafka or MapR Streams) as in Figure 6-2, it also helps to set you up for other projects, which is our Goal 2.

Multiuse Streaming Data

Almost as soon as a system like our fraud detector prototype is built, additional requirements will be added. For instance, whoever is charged with building the next-generation model for the fraud detector will want to analyze historical data. Other projects beyond fraud detection also may need to make use of the card transaction data.

A streams-based architecture makes it possible to design a system in which you can easily add additional services. In this case, the dataflow design for fraud detection based on card velocity makes the data about card activity more widely available, as depicted in Figure 6-3. In particular, exposing the card activity message queue to other internal services makes it easy for other services to see a comprehensive view of all authorization traffic. Remember that this data stream includes the decision results output by the fraud detector.

ndsa 0603
Figure 6-3. An advantage of using a message queue for the output of the fraud detector is shown here. This design lets us use the card activity and fraud decision data more broadly. Having a long retention time for data in the queue is also very useful. For example, that data could be queried by a process to build a collection of card location histories stored in a data format such as Parquet that allows very efficient querying. Other processes might use data from the card activity queue to determine the rate of fraud among subscribers or to build a real-time display of where transactions are happening.

Scaling Up the Fraud Detector

We just saw how a streaming architecture allows us to build a simple fraud detection prototype that also allows other services to access data so that historical analyses can improve the decision model or so that real-time displays can show us what is happening in the moment.

A streaming architecture helps in other ways as well. Figure 6-4 shows how multiple fraud detectors can use the same card activity message queue.

ndsa 0604
Figure 6-4. Multiple fraud detectors can use the same message queue. This design enables each of them to use all of the card activity data and thus be able to handle any incoming transaction.

The idea is that since all fraud models send all of their transactions to the card activity message queue, that queue will have a complete view of all card activity. Each instance of the fraud detector will not depend on any of the other instances, except insofar as more data will appear in each local database. The decoupling between services inherent in the streaming design allows the fraud service to be scaled relatively easily. Similarly, because each instance of the fraud detector maintains its own database of last card usage locations, new instances can be fielded that use alternative kinds of databases. If the number of transactions per second grows high enough that the current version of the detector cannot handle it, then an in-memory database can be fitted instead. Alternatively, the card activity message queue can use a partitioned topic so each fraud detector can be designed to handle only one partition.


In this fraud detection example, we’ve seen how the use of a message stream can make data available to more than one service without creating dependencies. In this way, the universal streaming-first design provides a powerful and flexible system that supports the microservices approach. For best results, it should become a habit in how you think about architectures that best suit your purposes.