This is a summary of book Design data intensive application - chap 1.

For anyone interested but don't have much of time.

Overview

A data intensive application, system often requires:

1. Store to retrieval effectively (DB, search index)

2. Send a message to another process, to be handled asynchronously (stream processing)

3. Remember the result of an expensive operation, to speed up reads (caches)

4. Periodically crunch a large amount of accumulated data (batch processing)

Reliability

Always function properly, withstand fault from:

1. Human:

   1. Design to minimizes opportunities for error making, especially from UI.

   2. Decouple the places where people make the most mistakes from the places where they can cause failure.

   3. Test by unit test, integration test.

   4. Have quick-easy recovery method.

   5. Set up monitoring, especially for performance and error rate.

2. Hardware

3. Software: cascading failures spread out services, exception crash process

4. Prevent abuse, unauthorized action.

Scalability

Scalability is a system ability, to decrease or increase in performance and cost on specific changing demands, load traffic...

1. Describe load params: Ex:

   1. Requests per second to a web server

   2. The ratio of reads to writes in a database

   3. Qty of simultaneously active users in a chat room...

   4. Data volume, traffic volume, throughput

2. Then measure performance, usually:

   1. Duration: short resp time, short data transmission time. Fast data (de)serialization, (de)compression => avoid latency.

   2. Space: low resource utilization such as CPU, RAM, bandwidth, IOPS...

3. Can classify performance onto 2 types:

   1. Macro performance (design lv)

   2. Micro performance (fine tune lv): about complexity of the algorithm, be careful premature optimization, mostly you can optimize something small and degrade something bigger.

   3. System will grow up in load params, complexity.... A well-scale system should be able to handle it well. Ask:

      1. What if load params increase, and you keep resource?

      2. If I want to keep performance, how to change system resource?

4. Usually treat a measurement number as a distribution, leverage percentile. Ex: the distribution of an API response, percentile 95% represent some rare cases that this API is too slow.

5. Notice idea about bottle neck: a parallel backend call, the lowest will adversely affect the rest.

Maintainability

1. Operate, read, add more feature, fix bug... easily.