Who Should Read This Book?

If you develop applications that have some kind of server/backend for storing or processing data, and your applications use the internet (e.g., web applications, mobile apps, or internet-connected sensors), then this book is for you.

如果您开发的应用程序有一些服务器/后端来存储或处理数据，并且您的应用程序使用互联网（例如网络应用程序、移动应用程序或连接到互联网的传感器），那么这本书适合您。

This book is for software engineers, software architects, and technical managers who love to code. It is especially relevant if you need to make decisions about the architecture of the systems you work on—for example, if you need to choose tools for solving a given problem and figure out how best to apply them. But even if you have no choice over your tools, this book will help you better understand their strengths and weaknesses.

这本书适合软件工程师、软件架构师和技术经理，他们热爱编程。 如果您需要对系统架构做出决策，例如选择解决特定问题的工具以及最佳应用它们，这本书尤为相关。但即使您对工具没有选择权，这本书也可以帮助您更好地了解它们的优势和劣势。

You should have some experience building web-based applications or network services, and you should be familiar with relational databases and SQL. Any non-relational databases and other data-related tools you know are a bonus, but not required. A general understanding of common network protocols like TCP and HTTP is helpful. Your choice of programming language or framework makes no difference for this book.

你应该拥有一定的构建基于网络的应用程序或网络服务的经验，并且应该熟悉关系型数据库和SQL。你所了解的任何非关系型数据库和其他数据相关工具都是bonus，但不是必需的。对于像TCP和HTTP这样的常见网络协议的基本理解是有帮助的。在本书中，你选择的编程语言或框架没有任何区别。

If any of the following are true for you, you’ll find this book valuable:

如果以下情况适用于您，您会发现这本书非常有价值：

• You want to learn how to make data systems scalable, for example, to support web or mobile apps with millions of users.

  你想学习如何使数据系统可扩展，例如，支持拥有数百万用户的网站或移动应用程序。

• You need to make applications highly available (minimizing downtime) and operationally robust.

  您需要使应用程序高度可用（最小化停机时间）和操作上健壮。

• You are looking for ways of making systems easier to maintain in the long run, even as they grow and as requirements and technologies change.

  您正在寻找使系统在长期发展和需求和技术变化时更易于维护的方法。

• You have a natural curiosity for the way things work and want to know what goes on inside major websites and online services. This book breaks down the internals of various databases and data processing systems, and it’s great fun to explore the bright thinking that went into their design.

  你对事物的运作方式有天生的好奇心，想要了解主要网站和在线服务内部的工作原理。这本书剖析了各种数据库和数据处理系统的内部结构，探索其设计背后的精妙思考过程，非常有趣。

Sometimes, when discussing scalable data systems, people make comments along the lines of, “You’re not Google or Amazon. Stop worrying about scale and just use a relational database.” There is truth in that statement: building for scale that you don’t need is wasted effort and may lock you into an inflexible design. In effect, it is a form of premature optimization. However, it’s also important to choose the right tool for the job, and different technologies each have their own strengths and weaknesses. As we shall see, relational databases are important but not the final word on dealing with data.

有时候，当讨论可扩展数据系统时，人们会做出类似的评论："你不是谷歌或亚马逊。别担心规模，只需使用关系数据库"。 这种说法确实有一定道理：为未来的规模而建造是浪费的，可能会使设计变得不灵活，实际上是一种过早的优化。 然而，选择正确的工具也同样重要，不同的技术各有其优点和缺点。正如我们所看到的，关系数据库很重要，但并不是解决处理数据的最终方案。

Scope of This Book

This book does not attempt to give detailed instructions on how to install or use specific software packages or APIs, since there is already plenty of documentation for those things. Instead we discuss the various principles and trade-offs that are fundamental to data systems, and we explore the different design decisions taken by different products.

本书无意提供如何安装或使用特定软件包或API的详细说明，因为这些内容已经有了大量的文档。相反，我们讨论了数据系统中基本的原则和权衡，探讨了不同产品所采取的不同设计决策。

In the ebook editions we have included links to the full text of online resources. All links were verified at the time of publication, but unfortunately links tend to break frequently due to the nature of the web. If you come across a broken link, or if you are reading a print copy of this book, you can look up references using a search engine. For academic papers, you can search for the title in Google Scholar to find open-access PDF files. Alternatively, you can find all of the references at https://github.com/ept/ddia-references , where we maintain up-to-date links.

在电子书版本中，我们已经包含了指向在线资源全文的链接。所有链接在出版时都经过验证，但不幸的是，由于网络的本质，链接容易中断。如果您遇到破损的链接，或者正在阅读本书的印刷版，则可以使用搜索引擎查找参考资料。对于学术论文，您可以在Google学术中搜索标题以找到开放获取的PDF文件。或者，您可以在https://github.com/ept/ddia-references找到所有参考资料的最新链接。

We look primarily at the architecture of data systems and the ways they are integrated into data-intensive applications. This book doesn’t have space to cover deployment, operations, security, management, and other areas—those are complex and important topics, and we wouldn’t do them justice by making them superficial side notes in this book. They deserve books of their own.

我们主要关注数据系统的架构及其集成进数据密集型应用程序中的方式。本书不涉及部署，运营，安全，管理和其他领域 - 这些都是复杂且重要的主题，我们不会在本书中简单地点到它们。它们都值得有一本专门的书来讲述它们。

Many of the technologies described in this book fall within the realm of the Big Data buzzword. However, the term “Big Data” is so overused and underdefined that it is not useful in a serious engineering discussion. This book uses less ambiguous terms, such as single-node versus distributed systems, or online/interactive versus offline/batch processing systems.

本书描述的许多技术都属于Big Data热词的范畴。然而，“Big Data”这个术语已被过度使用和缺乏定义，因此在严肃的工程讨论中并不有用。本书使用较少含糊的术语，如单节点与分布式系统，或在线/交互式与离线/批处理系统。

This book has a bias toward free and open source software (FOSS), because reading, modifying, and executing source code is a great way to understand how something works in detail. Open platforms also reduce the risk of vendor lock-in. However, where appropriate, we also discuss proprietary software (closed-source software, software as a service, or companies’ in-house software that is only described in literature but not released publicly).

这本书有偏向于自由和开源软件（FOSS），因为阅读、修改和执行源代码是了解事物详细工作原理的好方法。开放的平台也可以降低厂商锁定的风险。然而，在适当的情况下，我们也会讨论专有软件（闭源软件、软件即服务或公司内部的软件只在文献中描述但未公开发布）。

Outline of This Book

This book is arranged into three parts:

这本书分为三个部分：

1. In Part I , we discuss the fundamental ideas that underpin the design of data-intensive applications. We start in Chapter 1 by discussing what we’re actually trying to achieve: reliability, scalability, and maintainability; how we need to think about them; and how we can achieve them. In Chapter 2 we compare several different data models and query languages, and see how they are appropriate to different situations. In Chapter 3 we talk about storage engines: how databases arrange data on disk so that we can find it again efficiently. Chapter 4 turns to formats for data encoding (serialization) and evolution of schemas over time.

   在第一部分中，我们讨论了设计数据密集型应用程序所依托的基本思想。我们从第1章开始讨论我们实际要实现的目标：可靠性、可扩展性和可维护性；我们需要如何思考它们以及如何实现它们。在第2章中，我们比较了几种不同的数据模型和查询语言，并看到它们适用于不同的情况。在第3章中，我们讨论了存储引擎：数据库如何在磁盘上安排数据以便我们可以高效地找到它。第4章则转向数据编码格式（序列化）和模式随时间演变的格式。

2. In Part II , we move from data stored on one machine to data that is distributed across multiple machines. This is often necessary for scalability, but brings with it a variety of unique challenges. We first discuss replication ( Chapter 5 ), partitioning/sharding ( Chapter 6 ), and transactions ( Chapter 7 ). We then go into more detail on the problems with distributed systems ( Chapter 8 ) and what it means to achieve consistency and consensus in a distributed system ( Chapter 9 ).

   在第二部分，我们从存储于单独设备的数据转移到分布在多台设备上的数据。这通常是为了可伸缩性，但也带来了许多独特挑战。我们首先讨论复制（第五章），分区/分片（第六章）和事务（第七章）。然后，我们更详细地讨论分布式系统的问题（第八章），以及在分布式系统中实现一致性和共识的意义（第九章）。

3. In Part III , we discuss systems that derive some datasets from other datasets. Derived data often occurs in heterogeneous systems: when there is no one database that can do everything well, applications need to integrate several different databases, caches, indexes, and so on. In Chapter 10 we start with a batch processing approach to derived data, and we build upon it with stream processing in Chapter 11 . Finally, in Chapter 12 we put everything together and discuss approaches for building reliable, scalable, and maintainable applications in the future.

   第三部分中，我们讨论从其他数据集派生一些数据集的系统。派生数据通常出现在异构系统中：当没有一个数据库可以很好地完成所有任务时，应用程序需要集成几个不同的数据库、缓存、索引等等。在第10章中，我们从批处理方法开始处理派生数据，并在第11章中再次加以扩展处理。最后，在第12章中，我们将所有内容汇总，并讨论未来构建可靠、可扩展和可维护应用程序的方法。

References and Further Reading

Most of what we discuss in this book has already been said elsewhere in some form or another—in conference presentations, research papers, blog posts, code, bug trackers, mailing lists, and engineering folklore. This book summarizes the most important ideas from many different sources, and it includes pointers to the original literature throughout the text. The references at the end of each chapter are a great resource if you want to explore an area in more depth, and most of them are freely available online.

本书讨论的大部分内容在其他地方以某种形式或另一种方式已经过提及—在会议演示、研究论文、博客文章、代码、缺陷跟踪器、邮件列表和工程传说中。本书总结了许多不同来源的最重要的思想，且在整个文本中都包含有指向原始文献的指针。每章结尾的参考文献是深入探索一个区域的极佳资源，其中大多数免费在网上可获取。

O’Reilly Safari

Members have access to thousands of books, training videos, Learning Paths, interactive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others.

会员可访问来自250多个出版商的数千本图书、培训视频、学习路径、交互式教程和策划播放列表，包括O'Reilly Media、哈佛商业评论、Prentice Hall Professional、Addison-Wesley Professional、Microsoft Press、Sams、Que、Peachpit Press、Adobe、Focal Press、Cisco Press、John Wiley & Sons、Syngress、Morgan Kaufmann、IBM Redbooks、Packt、Adobe Press、FT Press、Apress、Manning、New Riders、McGraw-Hill、Jones＆Bartlett以及Course Technology等。

For more information, please visit http://oreilly.com/safari .

请访问http://oreilly.com/safari获取更多信息。

How to Contact Us

Please address comments and questions concerning this book to the publisher:

请将有关此书的评论和问题发送给出版社：

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/designing-data-intensive-apps .

我们有一本书的网页，列出勘误、例子和其他信息。您可以访问 http://bit.ly/designing-data-intensive-apps 查看此页。

To comment or ask technical questions about this book, send email to [email protected] .

如需对本书进行评论或提出技术问题，请发送电子邮件至[email protected]。

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com .

有关我们的图书、课程、会议和新闻的更多信息，请访问我们的网站http://www.oreilly.com。

Find us on Facebook: http://facebook.com/oreilly

在Facebook上关注我们：http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

关注我们的Twitter账户：http://twitter.com/oreillymedia。

Watch us on YouTube: http://www.youtube.com/oreillymedia

在YouTube上观看我们：http://www.youtube.com/oreillymedia。

Acknowledgments

This book is an amalgamation and systematization of a large number of other people’s ideas and knowledge, combining experience from both academic research and industrial practice. In computing we tend to be attracted to things that are new and shiny, but I think we have a huge amount to learn from things that have been done before. This book has over 800 references to articles, blog posts, talks, documentation, and more, and they have been an invaluable learning resource for me. I am very grateful to the authors of this material for sharing their knowledge.

这本书是对大量其他人的思想和知识进行的汇编和系统化，结合了学术研究和工业实践的经验。在计算领域，我们往往被新奇的东西所吸引，但我认为我们有很多东西可以从以前的事情中学习。这本书引用了800多篇文章、博客文章、讲话和文档等，它们对我来说是一种宝贵的学习资源。我非常感谢这些材料的作者分享他们的知识。

I have also learned a lot from personal conversations, thanks to a large number of people who have taken the time to discuss ideas or patiently explain things to me. In particular, I would like to thank Joe Adler, Ross Anderson, Peter Bailis, Márton Balassi, Alastair Beresford, Mark Callaghan, Mat Clayton, Patrick Collison, Sean Cribbs, Shirshanka Das, Niklas Ekström, Stephan Ewen, Alan Fekete, Gyula Fóra, Camille Fournier, Andres Freund, John Garbutt, Seth Gilbert, Tom Haggett, Pat Helland, Joe Hellerstein, Jakob Homan, Heidi Howard, John Hugg, Julian Hyde, Conrad Irwin, Evan Jones, Flavio Junqueira, Jessica Kerr, Kyle Kingsbury, Jay Kreps, Carl Lerche, Nicolas Liochon, Steve Loughran, Lee Mallabone, Nathan Marz, Caitie McCaffrey, Josie McLellan, Christopher Meiklejohn, Ian Meyers, Neha Narkhede, Neha Narula, Cathy O’Neil, Onora O’Neill, Ludovic Orban, Zoran Perkov, Julia Powles, Chris Riccomini, Henry Robinson, David Rosenthal, Jennifer Rullmann, Matthew Sackman, Martin Scholl, Amit Sela, Gwen Shapira, Greg Spurrier, Sam Stokes, Ben Stopford, Tom Stuart, Diana Vasile, Rahul Vohra, Pete Warden, and Brett Wooldridge.

因为许多人抽出时间与我探讨问题或耐心向我解释事物，我也从个人交流中学到了很多。特别感谢以下人士： Joe Adler， Ross Anderson， Peter Bailis， Márton Balassi， Alastair Beresford， Mark Callaghan， Mat Clayton， Patrick Collison， Sean Cribbs， Shirshanka Das， Niklas Ekström， Stephan Ewen， Alan Fekete， Gyula Fóra， Camille Fournier， Andres Freund， John Garbutt， Seth Gilbert， Tom Haggett， Pat Helland， Joe Hellerstein， Jakob Homan， Heidi Howard， John Hugg， Julian Hyde， Conrad Irwin， Evan Jones， Flavio Junqueira， Jessica Kerr， Kyle Kingsbury， Jay Kreps， Carl Lerche， Nicolas Liochon， Steve Loughran， Lee Mallabone， Nathan Marz， Caitie McCaffrey， Josie McLellan， Christopher Meiklejohn， Ian Meyers， Neha Narkhede， Neha Narula， Cathy O’Neil， Onora O’Neill， Ludovic Orban， Zoran Perkov， Julia Powles， Chris Riccomini， Henry Robinson， David Rosenthal， Jennifer Rullmann， Matthew Sackman， Martin Scholl， Amit Sela， Gwen Shapira， Greg Spurrier， Sam Stokes， Ben Stopford， Tom Stuart， Diana Vasile， Rahul Vohra， Pete Warden， 和 Brett Wooldridge.

Several more people have been invaluable to the writing of this book by reviewing drafts and providing feedback. For these contributions I am particularly indebted to Raul Agepati, Tyler Akidau, Mattias Andersson, Sasha Baranov, Veena Basavaraj, David Beyer, Jim Brikman, Paul Carey, Raul Castro Fernandez, Joseph Chow, Derek Elkins, Sam Elliott, Alexander Gallego, Mark Grover, Stu Halloway, Heidi Howard, Nicola Kleppmann, Stefan Kruppa, Bjorn Madsen, Sander Mak, Stefan Podkowinski, Phil Potter, Hamid Ramazani, Sam Stokes, and Ben Summers. Of course, I take all responsibility for any remaining errors or unpalatable opinions in this book.

这本书的写作离不开以下这些人的宝贵意见和反馈。我深表感激：Raul Agepati、Tyler Akidau、Mattias Andersson、Sasha Baranov、Veena Basavaraj、David Beyer、Jim Brikman、Paul Carey、Raul Castro Fernandez、Joseph Chow、Derek Elkins、Sam Elliott、Alexander Gallego、Mark Grover、Stu Halloway、Heidi Howard、Nicola Kleppmann、Stefan Kruppa、Bjorn Madsen、Sander Mak、Stefan Podkowinski、Phil Potter、Hamid Ramazani、Sam Stokes 和 Ben Summers。当然，本书中任何剩余的错误或不受欢迎的观点都由作者自负。

For helping this book become real, and for their patience with my slow writing and unusual requests, I am grateful to my editors Marie Beaugureau, Mike Loukides, Ann Spencer, and all the team at O’Reilly. For helping find the right words, I thank Rachel Head. For giving me the time and freedom to write in spite of other work commitments, I thank Alastair Beresford, Susan Goodhue, Neha Narkhede, and Kevin Scott.

感谢我的编辑玛丽·博戈罗、迈克·卢基德斯、安·斯宾塞以及O’Reilly的整个团队，他们帮助这本书成为现实，并耐心应对我的缓慢写作和不寻常的请求。感谢Rachel Head帮助我找到合适的词语。感谢Alastair Beresford、Susan Goodhue、Neha Narkhede和Kevin Scott给予我时间和自由，让我在其他工作任务的影响下完成写作。

Very special thanks are due to Shabbir Diwan and Edie Freedman, who illustrated with great care the maps that accompany the chapters. It’s wonderful that they took on the unconventional idea of creating maps, and made them so beautiful and compelling.

非常特别感谢沙比尔迪旺和埃迪弗里德曼，他们非常用心地插绘了随章节附带的地图。他们接受了非传统的创意，制作的地图美轮美奂，极富吸引力，实在太棒了！

Finally, my love goes to my family and friends, without whom I would not have been able to get through this writing process that has taken almost four years. You’re the best.

最后，我的爱送给我的家人和朋友们，没有你们的支持，我无法度过这长达四年的写作过程。你们是最棒的。