Hardware Faults

When we think of causes of system failure, hardware faults quickly come to mind. Hard disks crash, RAM becomes faulty, the power grid has a blackout, someone unplugs the wrong network cable. Anyone who has worked with large datacenters can tell you that these things happen all the time when you have a lot of machines.

当我们考虑系统故障的原因时，硬件故障很快就会跳入脑海。硬盘崩溃，内存损坏，电网停电，有人拔了错误的网络电缆。任何曾在大型数据中心工作过的人都可以告诉你，当你有很多机器时，这些事情经常发生。

Hard disks are reported as having a mean time to failure (MTTF) of about 10 to 50 years [ 5 , 6 ]. Thus, on a storage cluster with 10,000 disks, we should expect on average one disk to die per day.

硬盘通常被报告有一个平均故障时间（MTTF）约为10到50年[5, 6]。因此，在一个拥有10,000个硬盘的存储群集中，我们平均每天应该期望有一块硬盘损坏。

Our first response is usually to add redundancy to the individual hardware components in order to reduce the failure rate of the system. Disks may be set up in a RAID configuration, servers may have dual power supplies and hot-swappable CPUs, and datacenters may have batteries and diesel generators for backup power. When one component dies, the redundant component can take its place while the broken component is replaced. This approach cannot completely prevent hardware problems from causing failures, but it is well understood and can often keep a machine running uninterrupted for years.

我们通常的第一反应是将冗余添加到个别硬件组件中，以降低系统的故障率。可以将磁盘设置为RAID配置，服务器可以拥有双重电源和可热插拔的CPU，数据中心可以拥有备用电源的电池和柴油发电机。当一个组件出现故障时，冗余组件可以代替它的位置，而破损的组件则需要被更换。这种方法不能完全防止硬件问题导致故障，但是它是众所周知的，并且通常可以使一台机器连续运行多年。

Until recently, redundancy of hardware components was sufficient for most applications, since it makes total failure of a single machine fairly rare. As long as you can restore a backup onto a new machine fairly quickly, the downtime in case of failure is not catastrophic in most applications. Thus, multi-machine redundancy was only required by a small number of applications for which high availability was absolutely essential.

直到最近，硬件组件的冗余对于大多数应用程序来说已经足够，因为它使单台机器的完全失效变得相当罕见。只要能够相当快地将备份恢复到新机器上，在大多数应用程序中，发生故障时停机时间并不是灾难性的。因此，仅有少数应用程序需要多台机器的冗余，这对于绝对必要的高可用性是必需的。

However, as data volumes and applications’ computing demands have increased, more applications have begun using larger numbers of machines, which proportionally increases the rate of hardware faults. Moreover, in some cloud platforms such as Amazon Web Services (AWS) it is fairly common for virtual machine instances to become unavailable without warning [ 7 ], as the platforms are designed to prioritize flexibility and elasticity ⁱ over single-machine reliability.

然而，随着数据量和应用程序的计算需求增加，更多的应用程序开始使用更多的机器，这些机器的故障率也相应增加。此外，在一些云平台，例如AWS，虚拟机实例经常会在没有警告的情况下变得不可用[7]，因为这些平台的设计更注重灵活性和弹性而不是单机可靠性。

Hence there is a move toward systems that can tolerate the loss of entire machines, by using software fault-tolerance techniques in preference or in addition to hardware redundancy. Such systems also have operational advantages: a single-server system requires planned downtime if you need to reboot the machine (to apply operating system security patches, for example), whereas a system that can tolerate machine failure can be patched one node at a time, without downtime of the entire system (a rolling upgrade ; see Chapter 4 ).

因此，现在越来越倾向于使用软件容错技术来替代或补充硬件冗余，以实现整个机器的失效容忍系统。这样的系统还具有操作上的优势：单服务器系统需要计划停机时间才能重新启动机器（例如，应用操作系统安全补丁），而能够容忍机器故障的系统可以逐个升级节点而不需要整个系统停机（滚动升级；参见第四章）。

Software Errors

We usually think of hardware faults as being random and independent from each other: one machine’s disk failing does not imply that another machine’s disk is going to fail. There may be weak correlations (for example due to a common cause, such as the temperature in the server rack), but otherwise it is unlikely that a large number of hardware components will fail at the same time.

通常我们认为硬件故障是随机和相互独立的：一台机器的硬盘故障并不意味着另一台机器的硬盘会发生故障。可能存在一些弱相关性（例如由于共同的原因，例如服务器机架中的温度），但除此之外，大量硬件组件同时发生故障的可能性很小。

Another class of fault is a systematic error within the system [ 8 ]. Such faults are harder to anticipate, and because they are correlated across nodes, they tend to cause many more system failures than uncorrelated hardware faults [ 5 ]. Examples include:

另一类故障是系统内的系统性错误[8]。 这种故障很难预测，因为它们在节点之间存在相关性，因此它们往往比不相关的硬件故障导致更多的系统故障[5]。 例如：

• A software bug that causes every instance of an application server to crash when given a particular bad input. For example, consider the leap second on June 30, 2012, that caused many applications to hang simultaneously due to a bug in the Linux kernel [ 9 ].

  一个软件错误导致每个应用服务器在接收到特定的错误输入后都会崩溃。例如，考虑2012年6月30日的闰秒导致许多应用程序同时挂起，原因是Linux内核的错误[9]。

• A runaway process that uses up some shared resource—CPU time, memory, disk space, or network bandwidth.

  一个耗尽了一些共享资源的失控进程——CPU时间、内存、磁盘空间或网络带宽。

• A service that the system depends on that slows down, becomes unresponsive, or starts returning corrupted responses.

  系统依赖的服务变慢、无响应或返回错误响应。

• Cascading failures, where a small fault in one component triggers a fault in another component, which in turn triggers further faults [ 10 ].

  级联故障，一个组件的小故障会引发另一个组件的故障，接着又会触发更多的故障[10]。

The bugs that cause these kinds of software faults often lie dormant for a long time until they are triggered by an unusual set of circumstances. In those circumstances, it is revealed that the software is making some kind of assumption about its environment—and while that assumption is usually true, it eventually stops being true for some reason [ 11 ].

导致这些软件故障的错误通常会在很长一段时间内处于休眠状态，直到它们被一组不寻常的情况激活。在那种情况下，软件会揭示出它对环境做出某种假设——虽然这个假设通常是正确的，但它最终由于某种原因停止成立[11]。

There is no quick solution to the problem of systematic faults in software. Lots of small things can help: carefully thinking about assumptions and interactions in the system; thorough testing; process isolation; allowing processes to crash and restart; measuring, monitoring, and analyzing system behavior in production. If a system is expected to provide some guarantee (for example, in a message queue, that the number of incoming messages equals the number of outgoing messages), it can constantly check itself while it is running and raise an alert if a discrepancy is found [ 12 ].

软件系统存在系统性故障问题并非能立即解决。有许多小方法可以帮助解决：认真考虑系统中的假设和交互作用；进行彻底的测试；隔离处理；允许进程崩溃和重新启动；在生产中测量、监控和分析系统的行为。如果一个系统要提供某种保证（例如，在消息队列中，输入消息数等于输出消息数），它可以在运行时不断检查本身，并在发现差异时发出警报 [12]。

Human Errors

Humans design and build software systems, and the operators who keep the systems running are also human. Even when they have the best intentions, humans are known to be unreliable. For example, one study of large internet services found that configuration errors by operators were the leading cause of outages, whereas hardware faults (servers or network) played a role in only 10–25% of outages [ 13 ].

人类设计和构建软件系统，保持系统运行的操作员也是人类。即使他们有最好的意图，人类被证明是不可靠的。例如，一项针对大型互联网服务的研究发现，运营商的配置错误是停机的主要原因，而硬件故障（服务器或网络）仅在10-25%的停机中发挥作用[13]。

How do we make our systems reliable, in spite of unreliable humans? The best systems combine several approaches:

我们如何在人类不可靠的情况下使我们的系统可靠？最好的系统结合了几种方法：

• Design systems in a way that minimizes opportunities for error. For example, well-designed abstractions, APIs, and admin interfaces make it easy to do “the right thing” and discourage “the wrong thing.” However, if the interfaces are too restrictive people will work around them, negating their benefit, so this is a tricky balance to get right.

  以最小化错误机会的方式设计系统。例如，设计良好的抽象、API和管理界面可以使“正确的事情”变得容易，并避免“错误的事情”。然而，如果接口太严格，人们会绕过它们，抵消其好处，因此这是一个棘手的平衡问题。

• Decouple the places where people make the most mistakes from the places where they can cause failures. In particular, provide fully featured non-production sandbox environments where people can explore and experiment safely, using real data, without affecting real users.

  将人们经常犯错的地方与可能导致失败的地方分离开来。特别是，提供完整功能的非生产沙盒环境，人们可以在其中安全地探索和实验，使用真实数据，而不会影响真实用户。

• Test thoroughly at all levels, from unit tests to whole-system integration tests and manual tests [ 3 ]. Automated testing is widely used, well understood, and especially valuable for covering corner cases that rarely arise in normal operation.

  彻底测试各个层面，从单元测试到整个系统集成测试和手动测试[3]。自动化测试被广泛使用，被很好地理解，并且在覆盖在正常操作中很少出现的边缘情况时尤为有价值。

• Allow quick and easy recovery from human errors, to minimize the impact in the case of a failure. For example, make it fast to roll back configuration changes, roll out new code gradually (so that any unexpected bugs affect only a small subset of users), and provide tools to recompute data (in case it turns out that the old computation was incorrect).

  允许快速轻松地从人为错误中恢复，以最小化故障的影响。例如，快速回滚配置更改，逐步推出新代码（以便任何意外错误只影响一小部分用户），并提供重新计算数据的工具（以防旧计算结果不正确）。

• Set up detailed and clear monitoring, such as performance metrics and error rates. In other engineering disciplines this is referred to as telemetry . (Once a rocket has left the ground, telemetry is essential for tracking what is happening, and for understanding failures [ 14 ].) Monitoring can show us early warning signals and allow us to check whether any assumptions or constraints are being violated. When a problem occurs, metrics can be invaluable in diagnosing the issue.

  建立详细和清晰的监控，例如性能指标和错误率。在其他工程学科中，这被称为遥测。 （一旦火箭离开地面，遥测对于跟踪发生的情况和理解故障至关重要[14]。）监测可以向我们显示早期警告信号，并允许我们检查是否违反任何假设或约束。当问题发生时，指标可以非常有价值地诊断问题。

• Implement good management practices and training—a complex and important aspect, and beyond the scope of this book.

  实施良好的管理实践和培训——这是一个复杂而重要的方面，超出本书的范围。

How Important Is Reliability?

Reliability is not just for nuclear power stations and air traffic control software—more mundane applications are also expected to work reliably. Bugs in business applications cause lost productivity (and legal risks if figures are reported incorrectly), and outages of ecommerce sites can have huge costs in terms of lost revenue and damage to reputation.

可靠性不仅针对核电站和空中交通控制软件——更普通的应用程序同样需要保证可靠运行。商业应用程序中的漏洞会导致生产力下降（如果数字报告不正确，还会面临法律风险），电子商务网站的停机则会造成巨额损失，且声誉受损。

Even in “noncritical” applications we have a responsibility to our users. Consider a parent who stores all their pictures and videos of their children in your photo application [ 15 ]. How would they feel if that database was suddenly corrupted? Would they know how to restore it from a backup?

即使在“非关键”应用程序中，我们也有责任对我们的用户负责。考虑一个将他们所有的孩子的照片和视频存储在你的照片应用程序[15]中的父母。如果该数据库突然损坏，他们会感觉如何？他们知道如何从备份中恢复它吗？

There are situations in which we may choose to sacrifice reliability in order to reduce development cost (e.g., when developing a prototype product for an unproven market) or operational cost (e.g., for a service with a very narrow profit margin)—but we should be very conscious of when we are cutting corners.

在某些情况下，我们可能选择在可靠性方面做出牺牲，以降低开发成本（例如，针对未经证实的市场开发原型产品）或运营成本（例如，针对利润非常微薄的服务）——但我们应该非常清醒地意识到我们是在削减开销。

Describing Load

First, we need to succinctly describe the current load on the system; only then can we discuss growth questions (what happens if our load doubles?). Load can be described with a few numbers which we call load parameters . The best choice of parameters depends on the architecture of your system: it may be requests per second to a web server, the ratio of reads to writes in a database, the number of simultaneously active users in a chat room, the hit rate on a cache, or something else. Perhaps the average case is what matters for you, or perhaps your bottleneck is dominated by a small number of extreme cases.

首先，我们需要简洁地描述系统上的当前负载；只有这样我们才能讨论增长问题（如果我们的负载翻倍会发生什么？）。负载可以用一些数字来描述，我们称之为负载参数。参数的最佳选择取决于您系统的架构：可能是每秒钟对Web服务器的请求，数据库中读写比例，聊天室中的同时活跃用户数量，缓存命中率或其他内容。也许对您来说平均情况最重要，或者您的瓶颈由少数几个极端情况主导。

To make this idea more concrete, let’s consider Twitter as an example, using data published in November 2012 [ 16 ]. Two of Twitter’s main operations are:

为了使这个想法更具体化，让我们以Twitter为例，使用2012年11月发表的数据[16]。 Twitter的两个主要操作是：

Post tweet

A user can publish a new message to their followers (4.6k requests/sec on average, over 12k requests/sec at peak).

一个用户可以向他们的关注者发布一条新信息（平均每秒4.6k个请求，峰值时每秒超过12k个请求）。

Home timeline

A user can view tweets posted by the people they follow (300k requests/sec).

一个用户可以查看其关注者发布的推文（每秒300k次请求）。

Simply handling 12,000 writes per second (the peak rate for posting tweets) would be fairly easy. However, Twitter’s scaling challenge is not primarily due to tweet volume, but due to fan-out ⁱⁱ —each user follows many people, and each user is followed by many people. There are broadly two ways of implementing these two operations:

处理每秒12000次写入（发布推文的峰值速率）将相当容易。然而，Twitter的扩展挑战不主要是由于推文数量，而是由于扇出-每个用户都关注许多人，每个用户都被许多人关注。实现这两个操作通常有两种方式：

1. Posting a tweet simply inserts the new tweet into a global collection of tweets. When a user requests their home timeline, look up all the people they follow, find all the tweets for each of those users, and merge them (sorted by time). In a relational database like in Figure 1-2 , you could write a query such as:

   发布一条推文只会将其插入到全球推文集合中。当用户请求其主页时间线时，找出他们关注的所有人，找到每个用户的所有推文，并合并它们（按时间排序）。在关系型数据库中，可以编写查询语句，例如图1-2中的查询语句：

   SELECT tweets.*, users.* FROM tweets
     JOIN users   ON tweets.sender_id    = users.id
     JOIN follows ON follows.followee_id = users.id
     WHERE follows.follower_id = current_user

2. Maintain a cache for each user’s home timeline—like a mailbox of tweets for each recipient user (see Figure 1-3 ). When a user posts a tweet , look up all the people who follow that user, and insert the new tweet into each of their home timeline caches. The request to read the home timeline is then cheap, because its result has been computed ahead of time.

   为每个用户的主页时间线维护一个缓存——就像每个接受者用户的推文邮箱（参见图1-3）。当用户发布一条推文时，查找所有关注该用户的人，并将新的推文插入到他们每个人的主页时间线缓存中。因为结果已经提前计算出来，所以读取主页时间线的请求非常便宜。

Figure 1-2. Simple relational schema for implementing a Twitter home timeline.

Figure 1-3. Twitter’s data pipeline for delivering tweets to followers, with load parameters as of November 2012 [ 16 ].

The first version of Twitter used approach 1, but the systems struggled to keep up with the load of home timeline queries, so the company switched to approach 2. This works better because the average rate of published tweets is almost two orders of magnitude lower than the rate of home timeline reads, and so in this case it’s preferable to do more work at write time and less at read time.

Twitter的第一版使用方法1，但是那时系统很难跟上主页时间轴查询的负载，所以公司转而使用方法2。这个方法效果更好，因为发布推文的平均速率比主页时间轴读取的速率低了近两个数量级，所以在这种情况下，在写入时做更多的工作，而在读取时做更少的工作更好。

However, the downside of approach 2 is that posting a tweet now requires a lot of extra work. On average, a tweet is delivered to about 75 followers, so 4.6k tweets per second become 345k writes per second to the home timeline caches. But this average hides the fact that the number of followers per user varies wildly, and some users have over 30 million followers. This means that a single tweet may result in over 30 million writes to home timelines! Doing this in a timely manner—Twitter tries to deliver tweets to followers within five seconds—is a significant challenge.

然而，方法2的缺点是现在发布推文需要额外的工作量。平均而言，一条推文会被传递给约75个关注者，因此每秒4.6k条推文就变成了对主页时间轴缓存的345k次写入。但这个平均数隐藏了一个事实，即每个用户的关注者数量差异很大，有些用户拥有超过3000万的关注者。这意味着一条推文可能会导致3000万次对主页时间轴的写入！而在Twitter尝试在五秒内向关注者传递推文的情况下及时完成这项工作是一个巨大的挑战。

In the example of Twitter, the distribution of followers per user (maybe weighted by how often those users tweet) is a key load parameter for discussing scalability, since it determines the fan-out load. Your application may have very different characteristics, but you can apply similar principles to reasoning about its load.

以 Twitter 为例，用户的关注者分布（可能根据其推文频率加权）是讨论可扩展性的关键负载参数，因为它决定了扇出负载。你的应用程序可能具有非常不同的特点，但你可以应用类似的原则来推理它的负载。

The final twist of the Twitter anecdote: now that approach 2 is robustly implemented, Twitter is moving to a hybrid of both approaches. Most users’ tweets continue to be fanned out to home timelines at the time when they are posted, but a small number of users with a very large number of followers (i.e., celebrities) are excepted from this fan-out. Tweets from any celebrities that a user may follow are fetched separately and merged with that user’s home timeline when it is read, like in approach 1. This hybrid approach is able to deliver consistently good performance. We will revisit this example in Chapter 12 after we have covered some more technical ground.

发推特轶事的最后转折：现在实现了方法2，推特正在朝着两种方法的混合方式迈进。大多数用户的推特在发布时仍会在主页时间轴上扇出，但有一小部分拥有大量粉丝（即名人）的用户会被例外。用户可能会关注的任何名人的推特会单独获取并在读取时与该用户的主页时间轴合并，类似于方法1。这种混合方法能够提供一致的良好性能。在我们覆盖了一些更技术性的地面后，我们将在第12章重新审视这个例子。

Describing Performance

Once you have described the load on your system, you can investigate what happens when the load increases. You can look at it in two ways:

一旦你描述了系统的负载，你就可以调查当负载增加时会发生什么。你可以从两个方面来看待它：

• When you increase a load parameter and keep the system resources (CPU, memory, network bandwidth, etc.) unchanged, how is the performance of your system affected?

  当您增加负载参数并保持系统资源（CPU，内存，网络带宽等）不变时，系统的性能会受到影响吗？

• When you increase a load parameter, how much do you need to increase the resources if you want to keep performance unchanged?

  当你增加负载参数时，如果想保持性能不变，需要增加多少资源？

Both questions require performance numbers, so let’s look briefly at describing the performance of a system.

两个问题都需要性能数字，因此让我们简要地描述一下系统的性能。

In a batch processing system such as Hadoop, we usually care about throughput —the number of records we can process per second, or the total time it takes to run a job on a dataset of a certain size. ⁱⁱⁱ In online systems, what’s usually more important is the service’s response time —that is, the time between a client sending a request and receiving a response.

在像Hadoop这样的批处理系统中，我们通常关心吞吐量——每秒可以处理的记录数量，或者在特定大小的数据集上运行作业所需的总时间。在在线系统中，更重要的是服务的响应时间——即客户端发送请求和接收响应之间的时间。

Even if you only make the same request over and over again, you’ll get a slightly different response time on every try. In practice, in a system handling a variety of requests, the response time can vary a lot. We therefore need to think of response time not as a single number, but as a distribution of values that you can measure.

即使你一遍又一遍地做出相同的请求，每次尝试得到的响应时间也会稍有不同。在实践中，对于处理各种请求的系统，响应时间可能会有很大的差异。因此，我们需要将响应时间看作一组可以测量的值的分布，而不是一个单一的数字。

In Figure 1-4 , each gray bar represents a request to a service, and its height shows how long that request took. Most requests are reasonably fast, but there are occasional outliers that take much longer. Perhaps the slow requests are intrinsically more expensive, e.g., because they process more data. But even in a scenario where you’d think all requests should take the same time, you get variation: random additional latency could be introduced by a context switch to a background process, the loss of a network packet and TCP retransmission, a garbage collection pause, a page fault forcing a read from disk, mechanical vibrations in the server rack [ 18 ], or many other causes.

在图1-4中，每个灰色条代表对服务的请求，其高度显示了该请求所花费的时间。大多数请求的速度都比较快，但偶尔会出现一些耗时更长的异常值。也许慢请求本质上更昂贵，例如因为它们处理的数据更多。但即使在所有请求应该花费相同时间的情况下，你也会得到变化：随机额外的延迟可能会由于切换到后台进程、网络数据包的丢失和TCP重传、垃圾收集暂停、在服务器架中机械振动 [18] 或其他许多原因引入。

Figure 1-4. Illustrating mean and percentiles: response times for a sample of 100 requests to a service.

It’s common to see the average response time of a service reported. (Strictly speaking, the term “average” doesn’t refer to any particular formula, but in practice it is usually understood as the arithmetic mean : given n values, add up all the values, and divide by n .) However, the mean is not a very good metric if you want to know your “typical” response time, because it doesn’t tell you how many users actually experienced that delay.

通常会报告服务的平均响应时间。（严格来说，“平均值”一词并不指代任何特定的公式，但在实践中通常被理解为算术平均值：给定 n 个值，将所有值相加，然后除以 n。）然而，如果你想知道你的“典型”响应时间，平均数不是一个很好的度量标准，因为它不告诉你有多少个用户实际上经历了那个延迟。

Usually it is better to use percentiles . If you take your list of response times and sort it from fastest to slowest, then the median is the halfway point: for example, if your median response time is 200 ms, that means half your requests return in less than 200 ms, and half your requests take longer than that.

通常最好使用百分位数。如果您将响应时间列表按从最快到最慢进行排序，则中位数是中间点：例如，如果中位数响应时间为200毫秒，则表示您一半的请求在200毫秒以下返回，一半的请求需要更长时间。

This makes the median a good metric if you want to know how long users typically have to wait: half of user requests are served in less than the median response time, and the other half take longer than the median. The median is also known as the 50th percentile , and sometimes abbreviated as p50 . Note that the median refers to a single request; if the user makes several requests (over the course of a session, or because several resources are included in a single page), the probability that at least one of them is slower than the median is much greater than 50%.

这使得中位数成为一种良好的度量方法，如果您想知道用户通常需要等待多长时间：一半的用户请求在低于中位响应时间的情况下得到服务，另一半需要比中位数更长的时间。中位数也被称为第50个百分位数，并有时缩写为p50。请注意，中位数是指单个请求;如果用户发出多个请求（在会话期间或因为单个页面包括多个资源），则至少有一个请求慢于中位数的概率远大于50％。

In order to figure out how bad your outliers are, you can look at higher percentiles: the 95th , 99th , and 99.9th percentiles are common (abbreviated p95 , p99 , and p999 ). They are the response time thresholds at which 95%, 99%, or 99.9% of requests are faster than that particular threshold. For example, if the 95th percentile response time is 1.5 seconds, that means 95 out of 100 requests take less than 1.5 seconds, and 5 out of 100 requests take 1.5 seconds or more. This is illustrated in Figure 1-4 .

为了弄清异常数据有多严重，你可以查看更高的百分位数：95％、99％和99.9％百分位数是常见的（简写为p95、p99和p999）。它们是响应时间阈值，其中95％、99％或99.9％的请求速度比特定阈值更快。举例来说，如果95％分位数的响应时间为1.5秒，就意味着100个请求中有95个会在1.5秒内完成，而有5个需要1.5秒或更长时间。这在图1-4中有所说明。

High percentiles of response times, also known as tail latencies , are important because they directly affect users’ experience of the service. For example, Amazon describes response time requirements for internal services in terms of the 99.9th percentile, even though it only affects 1 in 1,000 requests. This is because the customers with the slowest requests are often those who have the most data on their accounts because they have made many purchases—that is, they’re the most valuable customers [ 19 ]. It’s important to keep those customers happy by ensuring the website is fast for them: Amazon has also observed that a 100 ms increase in response time reduces sales by 1% [ 20 ], and others report that a 1-second slowdown reduces a customer satisfaction metric by 16% [ 21 , 22 ].

高响应时间的百分位，也称为尾部延迟，非常重要，因为直接影响用户对服务的体验。例如，亚马逊在内部服务中以99.9百分位的响应时间要求为例，即使只影响了1,000个请求中的1个。这是因为最慢请求的客户通常是那些拥有许多账户数据的客户，因为他们进行了多次购买，即他们是最有价值的客户[19]。通过确保网站对这些客户快速，可以保持这些客户的满意度：亚马逊还观察到，响应时间增加100毫秒会导致销售额下降1%[20]，其他人则报告，1秒的减速会将客户满意度指标降低16% [21,22]。

On the other hand, optimizing the 99.99th percentile (the slowest 1 in 10,000 requests) was deemed too expensive and to not yield enough benefit for Amazon’s purposes. Reducing response times at very high percentiles is difficult because they are easily affected by random events outside of your control, and the benefits are diminishing.

另一方面，优化第99.99个百分位数（最慢的10000个请求中的1个）被认为对于亚马逊的目的来说太昂贵且利益不足。在非常高的百分位数上降低响应时间很困难，因为它们很容易受到无法控制的随机事件的影响，而且效益逐渐变小。

For example, percentiles are often used in service level objectives (SLOs) and service level agreements (SLAs), contracts that define the expected performance and availability of a service. An SLA may state that the service is considered to be up if it has a median response time of less than 200 ms and a 99th percentile under 1 s (if the response time is longer, it might as well be down), and the service may be required to be up at least 99.9% of the time. These metrics set expectations for clients of the service and allow customers to demand a refund if the SLA is not met.

例如，百分位数通常用于服务水平目标（SLO）和服务水平协议（SLA），这些协议定义了服务的预期性能和可用性。SLA 可以说明，如果服务的中位响应时间小于 200 毫秒且 99 百分位数小于 1 秒，则认为该服务处于上线状态（如果响应时间更长，则可以认为已经下线），并且该服务可能需要保持至少 99.9% 的可用性。这些指标为服务的客户设定了期望，并允许客户在未达到 SLA 的情况下要求退款。

Queueing delays often account for a large part of the response time at high percentiles. As a server can only process a small number of things in parallel (limited, for example, by its number of CPU cores), it only takes a small number of slow requests to hold up the processing of subsequent requests—an effect sometimes known as head-of-line blocking . Even if those subsequent requests are fast to process on the server, the client will see a slow overall response time due to the time waiting for the prior request to complete. Due to this effect, it is important to measure response times on the client side.

排队延迟通常占高百分位响应时间的很大一部分。由于一个服务器只能处理少量并行事项（例如受其CPU核心数量限制），只需要极少量缓慢请求就足以阻塞后续请求的处理——这种效应有时被称为头阻塞。即使这些后续请求在服务器上处理很快，客户端也会因等待先前请求完成而看到较慢的总响应时间。由于此效应，客户端测量响应时间非常重要。

When generating load artificially in order to test the scalability of a system, the load-generating client needs to keep sending requests independently of the response time. If the client waits for the previous request to complete before sending the next one, that behavior has the effect of artificially keeping the queues shorter in the test than they would be in reality, which skews the measurements [ 23 ].

当人工产生负载以测试系统的可扩展性时，产生负载的客户端需要独立于响应时间继续发送请求。如果客户端在发送下一个请求之前等待前一个请求完成，这种行为会在测试中人为地使队列比实际上要短，从而扭曲测量结果[23]。

Figure 1-5. When several backend calls are needed to serve a request, it takes just a single slow backend request to slow down the entire end-user request.

Approaches for Coping with Load

Now that we have discussed the parameters for describing load and metrics for measuring performance, we can start discussing scalability in earnest: how do we maintain good performance even when our load parameters increase by some amount?

现在我们已经讨论了描述负载的参数和测量性能的指标，我们可以认真讨论可扩展性了：即使我们的负载参数增加了一定量，我们如何保持良好的性能？

An architecture that is appropriate for one level of load is unlikely to cope with 10 times that load. If you are working on a fast-growing service, it is therefore likely that you will need to rethink your architecture on every order of magnitude load increase—or perhaps even more often than that.

如果一种架构适合某个负载水平，那么它很可能无法应对10倍于此的负载。如果你在开发一个快速增长的服务，那么在每一次负载增加一个数量级时，你很可能需要重新思考你的架构，甚至需要更频繁地重构。

People often talk of a dichotomy between scaling up ( vertical scaling , moving to a more powerful machine) and scaling out ( horizontal scaling , distributing the load across multiple smaller machines). Distributing load across multiple machines is also known as a shared-nothing architecture. A system that can run on a single machine is often simpler, but high-end machines can become very expensive, so very intensive workloads often can’t avoid scaling out. In reality, good architectures usually involve a pragmatic mixture of approaches: for example, using several fairly powerful machines can still be simpler and cheaper than a large number of small virtual machines.

人们经常谈论缩放的二分法（垂直缩放，移动到更强大的机器和水平缩放，将负载分布在多台较小的机器上）。在多台机器上分布负载也被称为“不共享任何东西”的架构。可以在单个机器上运行的系统通常更简单，但高端机器可能非常昂贵，因此非常密集的工作负载通常无法避免缩放。实际上，良好的架构通常涉及实用主义的方法混合：例如，使用几台相对强大的机器仍然可能比大量小虚拟机更简单，更便宜。

Some systems are elastic , meaning that they can automatically add computing resources when they detect a load increase, whereas other systems are scaled manually (a human analyzes the capacity and decides to add more machines to the system). An elastic system can be useful if load is highly unpredictable, but manually scaled systems are simpler and may have fewer operational surprises (see “Rebalancing Partitions” ).

一些系统是弹性的，意味着它们在检测到负载增加时可以自动添加计算资源，而其他系统则需要手动扩容（由人员分析容量并决定是否需要添加更多机器）。当负载高度不可预测时，弹性系统可能非常有用，但手动扩容的系统更简单，可能会有更少的操作意外（请参见“重新平衡分区”）。

While distributing stateless services across multiple machines is fairly straightforward, taking stateful data systems from a single node to a distributed setup can introduce a lot of additional complexity. For this reason, common wisdom until recently was to keep your database on a single node (scale up) until scaling cost or high-availability requirements forced you to make it distributed.

将无状态服务分发到多台机器上相对来说相当简单，但将有状态数据系统从单节点迁移至分布式环境则可能引入大量额外复杂性。因此，直到不久之前，通常被认为明智之举是将数据库保持在单一节点上（纵向扩展）直至扩展成本或高可用性需求迫使你将其变为分布式环境。

As the tools and abstractions for distributed systems get better, this common wisdom may change, at least for some kinds of applications. It is conceivable that distributed data systems will become the default in the future, even for use cases that don’t handle large volumes of data or traffic. Over the course of the rest of this book we will cover many kinds of distributed data systems, and discuss how they fare not just in terms of scalability, but also ease of use and maintainability.

随着分布式系统的工具和抽象层的不断完善，这些普遍的认识可能会改变，至少对某些应用而言是如此。未来，分布式数据系统有可能成为默认选择，即使是用于不处理大量数据或流量的案例。在本书的其余部分，我们将涵盖许多种分布式数据系统，并讨论它们在可扩展性、易用性和可维护性方面的表现。

The architecture of systems that operate at large scale is usually highly specific to the application—there is no such thing as a generic, one-size-fits-all scalable architecture (informally known as magic scaling sauce ). The problem may be the volume of reads, the volume of writes, the volume of data to store, the complexity of the data, the response time requirements, the access patterns, or (usually) some mixture of all of these plus many more issues.

大规模操作系统的架构通常高度特定于应用程序——没有通用的、一刀切的可扩展架构（俗称神奇扩展酱）。问题可能是读取量、写入量、存储数据量、数据复杂性、响应时间要求、访问模式，或（通常）所有这些问题的混合，加上许多其他问题。

For example, a system that is designed to handle 100,000 requests per second, each 1 kB in size, looks very different from a system that is designed for 3 requests per minute, each 2 GB in size—even though the two systems have the same data throughput.

例如，一个被设计为每秒处理100,000个1kB请求的系统，看起来与一个被设计为每分钟处理3个2GB请求的系统非常不同，即使这两个系统具有相同的数据吞吐量。

An architecture that scales well for a particular application is built around assumptions of which operations will be common and which will be rare—the load parameters. If those assumptions turn out to be wrong, the engineering effort for scaling is at best wasted, and at worst counterproductive. In an early-stage startup or an unproven product it’s usually more important to be able to iterate quickly on product features than it is to scale to some hypothetical future load.

一种可适用于特定应用的可扩展架构是基于哪些操作是常见的和哪些是罕见的假设——负载参数来构建的。如果这些假设被证明是错误的，那么扩展的工程努力将至少是浪费的，而在最坏的情况下可能是适得其反的。在初创阶段的创业公司或未经验证产品中，能够快速迭代产品特性通常比将来某个假设负载的扩展更加重要。

Even though they are specific to a particular application, scalable architectures are nevertheless usually built from general-purpose building blocks, arranged in familiar patterns. In this book we discuss those building blocks and patterns.

尽管可扩展架构是针对特定应用而设计的，但通常是由常见的通用构建块组成的，排列成熟悉的模式。在本书中，我们将讨论这些构建块和模式。

Operability: Making Life Easy for Operations

It has been suggested that “good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations” [ 12 ]. While some aspects of operations can and should be automated, it is still up to humans to set up that automation in the first place and to make sure it’s working correctly.

有人建议“好的操作通常可以克服不好（或不完整）的软件限制，但是好的软件不能可靠地运行不良操作”[12]。虽然某些操作方面可以和应该被自动化，但最终还是需要人类来进行第一次自动化设置，并确保其正确地运行。

Operations teams are vital to keeping a software system running smoothly. A good operations team typically is responsible for the following, and more [ 29 ]:

运维团队对于保持软件系统平稳运行非常重要。一个优秀的运维团队通常负责以下工作，以及更多其他事项 [29]：

• Monitoring the health of the system and quickly restoring service if it goes into a bad state

  监控系统的健康状况，如果系统状态不佳，迅速恢复服务。

• Tracking down the cause of problems, such as system failures or degraded performance

  追踪问题的原因，例如系统故障或性能降低。

• Keeping software and platforms up to date, including security patches

  保持软件和平台更新，包括安全补丁。

• Keeping tabs on how different systems affect each other, so that a problematic change can be avoided before it causes damage

  监控不同系统之间的相互影响，以避免出现可能导致破坏的问题性变化。

• Anticipating future problems and solving them before they occur (e.g., capacity planning)

  预料未来的问题并在它们发生之前解决它们（例如，容量规划）。

• Establishing good practices and tools for deployment, configuration management, and more

  建立好的实践和工具，用于部署、配置管理等方面。

• Performing complex maintenance tasks, such as moving an application from one platform to another

  执行复杂的维护任务，例如将一个应用程序从一个平台移动到另一个平台。

• Maintaining the security of the system as configuration changes are made

  随着配置变化的发生，维护系统的安全性。

• Defining processes that make operations predictable and help keep the production environment stable

  定义使操作可预测并帮助保持生产环境稳定的流程。

• Preserving the organization’s knowledge about the system, even as individual people come and go

  保留组织对系统的知识，即使个别人员进出。

Good operability means making routine tasks easy, allowing the operations team to focus their efforts on high-value activities. Data systems can do various things to make routine tasks easy, including:

良好的可操作性意味着使日常任务变得更加容易，使运营团队可以将精力集中在高价值的活动上。数据系统可以采取各种方式来简化日常任务，其中包括：

• Providing visibility into the runtime behavior and internals of the system, with good monitoring

  提供对系统运行行为和内部情况的可见性，带有良好的监控。

• Providing good support for automation and integration with standard tools

  为自动化提供良好支持并与标准工具进行集成。

• Avoiding dependency on individual machines (allowing machines to be taken down for maintenance while the system as a whole continues running uninterrupted)

  避免依赖特定的机器（使机器可以在维护期间关闭，而整个系统仍能无间断地运行）。

• Providing good documentation and an easy-to-understand operational model (“If I do X, Y will happen”)

  提供良好的文档和易于理解的操作模型（“如果我做 X，就会发生 Y”）。

• Providing good default behavior, but also giving administrators the freedom to override defaults when needed

  提供良好的默认行为，但也给管理员在需要时覆盖默认值的自由。

• Self-healing where appropriate, but also giving administrators manual control over the system state when needed

  在必要时进行自愈，但也在需要时赋予管理员手动控制系统状态的能力。

• Exhibiting predictable behavior, minimizing surprises

  展示可预测的行为，最小化惊喜。

Simplicity: Managing Complexity

Small software projects can have delightfully simple and expressive code, but as projects get larger, they often become very complex and difficult to understand. This complexity slows down everyone who needs to work on the system, further increasing the cost of maintenance. A software project mired in complexity is sometimes described as a big ball of mud [ 30 ].

小型软件项目可以拥有简单明了且表达清晰的代码，但随着项目规模的扩大，往往会变得非常复杂且难以理解。这种复杂性会拖慢每一个需要处理该系统的人，进一步增加了维护成本。一个陷入复杂困境的软件项目有时被描述为一个沉重的泥球。[30]

There are various possible symptoms of complexity: explosion of the state space, tight coupling of modules, tangled dependencies, inconsistent naming and terminology, hacks aimed at solving performance problems, special-casing to work around issues elsewhere, and many more. Much has been said on this topic already [ 31 , 32 , 33 ].

复杂性可能会出现各种症状：状态空间爆炸，模块之间紧密耦合，依赖关系错综复杂，命名和术语不一致，为解决性能问题而进行的修改，为解决其他问题而进行的特殊处理等等。已经有很多关于这个话题的讨论[31，32，33]。

When complexity makes maintenance hard, budgets and schedules are often overrun. In complex software, there is also a greater risk of introducing bugs when making a change: when the system is harder for developers to understand and reason about, hidden assumptions, unintended consequences, and unexpected interactions are more easily overlooked. Conversely, reducing complexity greatly improves the maintainability of software, and thus simplicity should be a key goal for the systems we build.

当复杂性使得维护困难时，预算和计划常常会超支。在复杂的软件中，更容易引入错误风险：当开发人员难以理解和推理系统时，隐藏的假设、意外的后果和意外的交互就很容易被忽视。相反，减少复杂性极大地提高了软件的可维护性，因此简单性应成为我们构建系统的关键目标。

Making a system simpler does not necessarily mean reducing its functionality; it can also mean removing accidental complexity. Moseley and Marks [ 32 ] define complexity as accidental if it is not inherent in the problem that the software solves (as seen by the users) but arises only from the implementation.

简化系统不一定意味着减少其功能；这也可能意味着消除偶然复杂性。Moseley和Marks [32]将复杂性定义为偶然的，如果它不是软件解决的问题本身固有的（如用户所看到的），而仅仅是由实现引起的。

One of the best tools we have for removing accidental complexity is abstraction . A good abstraction can hide a great deal of implementation detail behind a clean, simple-to-understand façade. A good abstraction can also be used for a wide range of different applications. Not only is this reuse more efficient than reimplementing a similar thing multiple times, but it also leads to higher-quality software, as quality improvements in the abstracted component benefit all applications that use it.

我们用来消除无意的复杂性的最佳工具之一是抽象化。良好的抽象化可以在一个干净、简单易懂的立面后面隐藏许多实现细节。良好的抽象化也可以用于广泛的不同应用。这种重用不仅比多次重新实现相似的东西更有效率，而且还会导致高质量的软件，因为在被抽象化的组件中的质量改进将使所有使用它的应用受益。

For example, high-level programming languages are abstractions that hide machine code, CPU registers, and syscalls. SQL is an abstraction that hides complex on-disk and in-memory data structures, concurrent requests from other clients, and inconsistencies after crashes. Of course, when programming in a high-level language, we are still using machine code; we are just not using it directly , because the programming language abstraction saves us from having to think about it.

例如，高级编程语言是抽象层，它们隐藏了机器码、CPU寄存器和系统调用。SQL是一个抽象层，它隐藏了复杂的磁盘和内存数据结构，其他客户端的并发请求，以及崩溃后的不一致性。当然，当使用高级语言编程时，我们仍然使用机器码；我们只是不直接使用它，因为编程语言的抽象层帮助我们不必考虑它。

However, finding good abstractions is very hard. In the field of distributed systems, although there are many good algorithms, it is much less clear how we should be packaging them into abstractions that help us keep the complexity of the system at a manageable level.

然而，寻找好的抽象概念是非常困难的。在分布式系统领域，虽然有许多好的算法，但我们如何将它们打包成有助于我们将系统的复杂性保持在可控范围内的抽象概念，却不太清楚。

Throughout this book, we will keep our eyes open for good abstractions that allow us to extract parts of a large system into well-defined, reusable components.

在本书中，我们将密切关注好的抽象，以便将大系统的部分提取到明确定义的可重用组件中。

Evolvability: Making Change Easy

It’s extremely unlikely that your system’s requirements will remain unchanged forever. They are much more likely to be in constant flux: you learn new facts, previously unanticipated use cases emerge, business priorities change, users request new features, new platforms replace old platforms, legal or regulatory requirements change, growth of the system forces architectural changes, etc.

你的系统需求永远不可能保持不变。它们更可能是不断变化的：你会学习到新的知识，出现以前未预料到的用例，业务重点会改变，用户会要求新的功能，新平台会取代旧平台，法律或监管要求会改变，系统的增长会迫使架构发生改变，等等。

In terms of organizational processes, Agile working patterns provide a framework for adapting to change. The Agile community has also developed technical tools and patterns that are helpful when developing software in a frequently changing environment, such as test-driven development (TDD) and refactoring.

在组织流程方面，敏捷工作模式提供了一个适应变化的框架。敏捷社区还开发了技术工具和模式，在频繁变化的环境中开发软件非常有帮助，例如测试驱动开发（TDD）和重构。

Most discussions of these Agile techniques focus on a fairly small, local scale (a couple of source code files within the same application). In this book, we search for ways of increasing agility on the level of a larger data system, perhaps consisting of several different applications or services with different characteristics. For example, how would you “refactor” Twitter’s architecture for assembling home timelines ( “Describing Load” ) from approach 1 to approach 2?

大多数关于这些敏捷技术的讨论都聚焦于相当小的、局部的范围（同一应用程序中的几个源代码文件）。在本书中，我们寻求在更大的数据系统层面上增加敏捷性的方法，这些系统可能包括多个具有不同特点的应用程序或服务。例如，您将如何将 Twitter 的首页时间线组装架构（“描述负载”）从方法1改进到方法2？

The ease with which you can modify a data system, and adapt it to changing requirements, is closely linked to its simplicity and its abstractions: simple and easy-to-understand systems are usually easier to modify than complex ones. But since this is such an important idea, we will use a different word to refer to agility on a data system level: evolvability [ 34 ].

你能够轻松修改和适应不断变化的需求的数据系统的能力，与其简单性和抽象性密切相关：易于理解的简单系统通常比复杂系统更容易修改。但由于这是一个非常重要的想法，因此我们将使用一个不同的词来指代数据系统层面的敏捷性：可发展性[34]。