Move several sections to Appendix
This commit is contained in:
Donne Martin
parent
28137d38f0
commit
a9cc1c0cbb
300
README.md
300
README.md
|
|
@ -159,6 +159,10 @@ Review the [Contributing Guidelines](https://github.com/donnemartin/system-desig
|
|||
* [Appendix](#appendix)
|
||||
* [Powers of two table](#powers-of-two-table)
|
||||
* [Latency numbers every programmer should know](#latency-numbers-every-programmer-should-know)
|
||||
* [Additional system design interview questions](#additional-system-design-interview-questions)
|
||||
* [Real world architectures](#real-world-architectures)
|
||||
* [Company architectures](#company-architectures)
|
||||
* [Company engineering blogs](#company-engineering-blogs)
|
||||
* [Under development](#under-development)
|
||||
* [Credits](#credits)
|
||||
* [Contact info](#contact-info)
|
||||
|
|
@ -339,154 +343,6 @@ Check out the following links to get a better idea of what to expect:
|
|||
| Design a circular array | [Contribute](#contributing) |
|
||||
| Add an object-oriented design question | [Contribute](#contributing) |
|
||||
|
||||
## Additional system design interview questions
|
||||
|
||||
> Common system design interview questions, with links to resources on how to solve each.
|
||||
|
||||
| Question | Reference(s) |
|
||||
|---|---|
|
||||
| Design a file sync service like Dropbox | [youtube.com](https://www.youtube.com/watch?v=PE4gwstWhmc) |
|
||||
| Design a search engine like Google | [queue.acm.org](http://queue.acm.org/detail.cfm?id=988407)<br/>[stackexchange.com](http://programmers.stackexchange.com/questions/38324/interview-question-how-would-you-implement-google-search)<br/>[ardendertat.com](http://www.ardendertat.com/2012/01/11/implementing-search-engines/)<br>[stanford.edu](http://infolab.stanford.edu/~backrub/google.html) |
|
||||
| Design a scalable web crawler like Google | [quora.com](https://www.quora.com/How-can-I-build-a-web-crawler-from-scratch) |
|
||||
| Design Google docs | [code.google.com](https://code.google.com/p/google-mobwrite/)<br/>[neil.fraser.name](https://neil.fraser.name/writing/sync/) |
|
||||
| Design a key-value store like Redis | [slideshare.net](http://www.slideshare.net/dvirsky/introduction-to-redis) |
|
||||
| Design a cache system like Memcached | [slideshare.net](http://www.slideshare.net/oemebamo/introduction-to-memcached) |
|
||||
| Design a recommendation system like Amazon's | [hulu.com](http://tech.hulu.com/blog/2011/09/19/recommendation-system.html)<br/>[ijcai13.org](http://ijcai13.org/files/tutorial_slides/td3.pdf) |
|
||||
| Design a tinyurl system like Bitly | [n00tc0d3r.blogspot.com](http://n00tc0d3r.blogspot.com/) |
|
||||
| Design a chat app like WhatsApp | [highscalability.com](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html)
|
||||
| Design a picture sharing system like Instagram | [highscalability.com](http://highscalability.com/flickr-architecture)<br/>[highscalability.com](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html) |
|
||||
| Design the Facebook news feed function | [quora.com](http://www.quora.com/What-are-best-practices-for-building-something-like-a-News-Feed)<br/>[quora.com](http://www.quora.com/Activity-Streams/What-are-the-scaling-issues-to-keep-in-mind-while-developing-a-social-network-feed)<br/>[slideshare.net](http://www.slideshare.net/danmckinley/etsy-activity-feeds-architecture) |
|
||||
| Design the Facebook timeline function | [facebook.com](https://www.facebook.com/note.php?note_id=10150468255628920)<br/>[highscalability.com](http://highscalability.com/blog/2012/1/23/facebook-timeline-brought-to-you-by-the-power-of-denormaliza.html) |
|
||||
| Design the Facebook chat function | [erlang-factory.com](http://www.erlang-factory.com/upload/presentations/31/EugeneLetuchy-ErlangatFacebook.pdf)<br/>[facebook.com](https://www.facebook.com/note.php?note_id=14218138919&id=9445547199&index=0) |
|
||||
| Design a graph search function like Facebook's | [facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-building-out-the-infrastructure-for-graph-search/10151347573598920)<br/>[facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-indexing-and-ranking-in-graph-search/10151361720763920)<br/>[facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-the-natural-language-interface-of-graph-search/10151432733048920) |
|
||||
| Design a content delivery network like CloudFlare | [cmu.edu](http://repository.cmu.edu/cgi/viewcontent.cgi?article=2112&context=compsci) |
|
||||
| Design a trending topic system like Twitter's | [michael-noll.com](http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/)<br/>[snikolov .wordpress.com](http://snikolov.wordpress.com/2012/11/14/early-detection-of-twitter-trends/) |
|
||||
| Design a random ID generation system | [blog.twitter.com](https://blog.twitter.com/2010/announcing-snowflake)<br/>[github.com](https://github.com/twitter/snowflake/) |
|
||||
| Return the top k requests during a time interval | [ucsb.edu](https://icmi.cs.ucsb.edu/research/tech_reports/reports/2005-23.pdf)<br/>[wpi.edu](http://davis.wpi.edu/xmdv/docs/EDBT11-diyang.pdf) |
|
||||
| Design a system that serves data from multiple data centers | [highscalability.com](http://highscalability.com/blog/2009/8/24/how-google-serves-data-from-multiple-datacenters.html) |
|
||||
| Design an online multiplayer card game | [indieflashblog.com](http://www.indieflashblog.com/how-to-create-an-asynchronous-multiplayer-game.html)<br/>[buildnewgames.com](http://buildnewgames.com/real-time-multiplayer/) |
|
||||
| Design a garbage collection system | [stuffwithstuff.com](http://journal.stuffwithstuff.com/2013/12/08/babys-first-garbage-collector/)<br/>[washington.edu](http://courses.cs.washington.edu/courses/csep521/07wi/prj/rick.pdf) |
|
||||
| Add a system design question | [Contribute](#contributing) |
|
||||
|
||||
## Real world architectures
|
||||
|
||||
> Articles on how real world systems are designed.
|
||||
|
||||
<p align="center">
|
||||
<img src="http://i.imgur.com/TcUo2fw.png">
|
||||
<br/>
|
||||
<i><a href=https://www.infoq.com/presentations/Twitter-Timeline-Scalability>Source: Twitter timelines at scale</a></i>
|
||||
</p>
|
||||
|
||||
**Don't focus on nitty gritty details for the following articles, instead:**
|
||||
|
||||
* Identify shared principles, common technologies, and patterns within these articles
|
||||
* Study what problems are solved by each component, where it works, where it doesn't
|
||||
* Review the lessons learned
|
||||
|
||||
|Type | System | Reference(s) |
|
||||
|---|---|---|
|
||||
| Data processing | **MapReduce** - Distributed data processing from Google | [research.google.com](http://static.googleusercontent.com/media/research.google.com/zh-CN/us/archive/mapreduce-osdi04.pdf) |
|
||||
| Data processing | **Spark** - Distributed data processing from Databricks | [slideshare.net](http://www.slideshare.net/AGrishchenko/apache-spark-architecture) |
|
||||
| Data processing | **Storm** - Distributed data processing from Twitter | [slideshare.net](http://www.slideshare.net/previa/storm-16094009) |
|
||||
| | | |
|
||||
| Data store | **Bigtable** - Distributed column-oriented database from Google | [harvard.edu](http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/chang06bigtable.pdf) |
|
||||
| Data store | **HBase** - Open source implementation of Bigtable | [slideshare.net](http://www.slideshare.net/alexbaranau/intro-to-hbase) |
|
||||
| Data store | **Cassandra** - Distributed column-oriented database from Facebook | [slideshare.net](http://www.slideshare.net/planetcassandra/cassandra-introduction-features-30103666)
|
||||
| Data store | **DynamoDB** - Document-oriented database from Amazon | [harvard.edu](http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf) |
|
||||
| Data store | **MongoDB** - Document-oriented database | [slideshare.net](http://www.slideshare.net/mdirolf/introduction-to-mongodb) |
|
||||
| Data store | **Spanner** - Globally-distributed database from Google | [research.google.com](http://research.google.com/archive/spanner-osdi2012.pdf) |
|
||||
| Data store | **Memcached** - Distributed memory caching system | [slideshare.net](http://www.slideshare.net/oemebamo/introduction-to-memcached) |
|
||||
| Data store | **Redis** - Distributed memory caching system with persistence and value types | [slideshare.net](http://www.slideshare.net/dvirsky/introduction-to-redis) |
|
||||
| | | |
|
||||
| File system | **Google File System (GFS)** - Distributed file system | [research.google.com](http://static.googleusercontent.com/media/research.google.com/zh-CN/us/archive/gfs-sosp2003.pdf) |
|
||||
| File system | **Hadoop File System (HDFS)** - Open source implementation of GFS | [apache.org](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) |
|
||||
| | | |
|
||||
| Misc | **Chubby** - Lock service for loosely-coupled distributed systems from Google | [research.google.com](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/chubby-osdi06.pdf) |
|
||||
| Misc | **Dapper** - Distributed systems tracing infrastructure | [research.google.com](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36356.pdf)
|
||||
| Misc | **Kafka** - Pub/sub message queue from LinkedIn | [slideshare.net](http://www.slideshare.net/mumrah/kafka-talk-tri-hug) |
|
||||
| Misc | **Zookeeper** - Centralized infrastructure and services enabling synchronization | [slideshare.net](http://www.slideshare.net/sauravhaloi/introduction-to-apache-zookeeper) |
|
||||
| | Add an architecture | [Contribute](#contributing) |
|
||||
|
||||
### Company architectures
|
||||
|
||||
| Company | Reference(s) |
|
||||
|---|---|
|
||||
| Amazon | [Amazon architecture](http://highscalability.com/amazon-architecture) |
|
||||
| Cinchcast | [Producing 1,500 hours of audio every day](http://highscalability.com/blog/2012/7/16/cinchcast-architecture-producing-1500-hours-of-audio-every-d.html) |
|
||||
| DataSift | [Realtime datamining At 120,000 tweets per second](http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html) |
|
||||
| DropBox | [How we've scaled Dropbox](https://www.youtube.com/watch?v=PE4gwstWhmc) |
|
||||
| ESPN | [Operating At 100,000 duh nuh nuhs per second](http://highscalability.com/blog/2013/11/4/espns-architecture-at-scale-operating-at-100000-duh-nuh-nuhs.html) |
|
||||
| Google | [Google architecture](http://highscalability.com/google-architecture) |
|
||||
| Instagram | [14 million users, terabytes of photos](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html)<br/>[What powers Instagram](http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances) |
|
||||
| Justin.tv | [Justin.Tv's live video broadcasting architecture](http://highscalability.com/blog/2010/3/16/justintvs-live-video-broadcasting-architecture.html) |
|
||||
| Facebook | [Scaling memcached at Facebook](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/key-value/fb-memcached-nsdi-2013.pdf)<br/>[TAO: Facebook’s distributed data store for the social graph](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/data-store/tao-facebook-distributed-datastore-atc-2013.pdf)<br/>[Facebook’s photo storage](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf) |
|
||||
| Flickr | [Flickr architecture](http://highscalability.com/flickr-architecture) |
|
||||
| Mailbox | [From 0 to one million users in 6 weeks](http://highscalability.com/blog/2013/6/18/scaling-mailbox-from-0-to-one-million-users-in-6-weeks-and-1.html) |
|
||||
| Pinterest | [From 0 To 10s of billions of page views a month](http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html)<br/>[18 million visitors, 10x growth, 12 employees](http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html) |
|
||||
| Playfish | [50 million monthly users and growing](http://highscalability.com/blog/2010/9/21/playfishs-social-gaming-architecture-50-million-monthly-user.html) |
|
||||
| PlentyOfFish | [PlentyOfFish architecture](http://highscalability.com/plentyoffish-architecture) |
|
||||
| Salesforce | [How they handle 1.3 billion transactions a day](http://highscalability.com/blog/2013/9/23/salesforce-architecture-how-they-handle-13-billion-transacti.html) |
|
||||
| Stack Overflow | [Stack Overflow architecture](http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html) |
|
||||
| TripAdvisor | [40M visitors, 200M dynamic page views, 30TB data](http://highscalability.com/blog/2011/6/27/tripadvisor-architecture-40m-visitors-200m-dynamic-page-view.html) |
|
||||
| Tumblr | [15 billion page views a month](http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html) |
|
||||
| Twitter | [Making Twitter 10000 percent faster](http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster)<br/>[Storing 250 million tweets a day using MySQL](http://highscalability.com/blog/2011/12/19/how-twitter-stores-250-million-tweets-a-day-using-mysql.html)<br/>[150M active users, 300K QPS, a 22 MB/S firehose](http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html)<br/>[Timelines at scale](https://www.infoq.com/presentations/Twitter-Timeline-Scalability)<br/>[Big and small data at Twitter](https://www.youtube.com/watch?v=5cKTP36HVgI)<br/>[Operations at Twitter: scaling beyond 100 million users](https://www.youtube.com/watch?v=z8LU0Cj6BOU) |
|
||||
| Uber | [How Uber scales their real-time market platform](http://highscalability.com/blog/2015/9/14/how-uber-scales-their-real-time-market-platform.html) |
|
||||
| WhatsApp | [The WhatsApp architecture Facebook bought for $19 billion](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html) |
|
||||
| YouTube | [YouTube scalability](https://www.youtube.com/watch?v=w5WVu624fY8)<br/>[YouTube architecture](http://highscalability.com/youtube-architecture) |
|
||||
|
||||
## Company engineering blogs
|
||||
|
||||
> Architectures for companies you are interviewing with.
|
||||
>
|
||||
> Questions you encounter might be from the same domain.
|
||||
|
||||
* [Airbnb Engineering](http://nerds.airbnb.com/)
|
||||
* [Atlassian Developers](https://developer.atlassian.com/blog/)
|
||||
* [Autodesk Engineering](http://cloudengineering.autodesk.com/blog/)
|
||||
* [AWS Blog](https://aws.amazon.com/blogs/aws/)
|
||||
* [Bitly Engineering Blog](http://word.bitly.com/)
|
||||
* [Box Blogs](https://www.box.com/blog/engineering/)
|
||||
* [Cloudera Developer Blog](http://blog.cloudera.com/blog/)
|
||||
* [Dropbox Tech Blog](https://tech.dropbox.com/)
|
||||
* [Engineering at Quora](http://engineering.quora.com/)
|
||||
* [Ebay Tech Blog](http://www.ebaytechblog.com/)
|
||||
* [Evernote Tech Blog](https://blog.evernote.com/tech/)
|
||||
* [Etsy Code as Craft](http://codeascraft.com/)
|
||||
* [Facebook Engineering](https://www.facebook.com/Engineering)
|
||||
* [Flickr Code](http://code.flickr.net/)
|
||||
* [Foursquare Engineering Blog](http://engineering.foursquare.com/)
|
||||
* [GitHub Engineering Blog](http://githubengineering.com/)
|
||||
* [Google Research Blog](http://googleresearch.blogspot.com/)
|
||||
* [Groupon Engineering Blog](https://engineering.groupon.com/)
|
||||
* [Heroku Engineering Blog](https://engineering.heroku.com/)
|
||||
* [Hubspot Engineering Blog](http://product.hubspot.com/blog/topic/engineering)
|
||||
* [High Scalability](http://highscalability.com/)
|
||||
* [Instagram Engineering](http://instagram-engineering.tumblr.com/)
|
||||
* [Intel Software Blog](https://software.intel.com/en-us/blogs/)
|
||||
* [Jane Street Tech Blog](https://blogs.janestreet.com/category/ocaml/)
|
||||
* [LinkedIn Engineering](http://engineering.linkedin.com/blog)
|
||||
* [Microsoft Engineering](https://engineering.microsoft.com/)
|
||||
* [Microsoft Python Engineering](https://blogs.msdn.microsoft.com/pythonengineering/)
|
||||
* [Netflix Tech Blog](http://techblog.netflix.com/)
|
||||
* [Paypal Developer Blog](https://devblog.paypal.com/category/engineering/)
|
||||
* [Pinterest Engineering Blog](http://engineering.pinterest.com/)
|
||||
* [Quora Engineering](https://engineering.quora.com/)
|
||||
* [Reddit Blog](http://www.redditblog.com/)
|
||||
* [Salesforce Engineering Blog](https://developer.salesforce.com/blogs/engineering/)
|
||||
* [Slack Engineering Blog](https://slack.engineering/)
|
||||
* [Spotify Labs](https://labs.spotify.com/)
|
||||
* [Twilio Engineering Blog](http://www.twilio.com/engineering)
|
||||
* [Twitter Engineering](https://engineering.twitter.com/)
|
||||
* [Uber Engineering Blog](http://eng.uber.com/)
|
||||
* [Yahoo Engineering Blog](http://yahooeng.tumblr.com/)
|
||||
* [Yelp Engineering Blog](http://engineeringblog.yelp.com/)
|
||||
* [Zynga Engineering Blog](https://www.zynga.com/blogs/engineering)
|
||||
|
||||
### Source(s) and further reading
|
||||
|
||||
* [kilimchoi/engineering-blogs](https://github.com/kilimchoi/engineering-blogs)
|
||||
|
||||
## System design topics: start here
|
||||
|
||||
New to system design?
|
||||
|
|
@ -1711,6 +1567,154 @@ Handy metrics based on numbers above:
|
|||
* [Designs, lessons, and advice from building large distributed systems](http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf)
|
||||
* [Software Engineering Advice from Building Large-Scale Distributed Systems](https://static.googleusercontent.com/media/research.google.com/en//people/jeff/stanford-295-talk.pdf)
|
||||
|
||||
### Additional system design interview questions
|
||||
|
||||
> Common system design interview questions, with links to resources on how to solve each.
|
||||
|
||||
| Question | Reference(s) |
|
||||
|---|---|
|
||||
| Design a file sync service like Dropbox | [youtube.com](https://www.youtube.com/watch?v=PE4gwstWhmc) |
|
||||
| Design a search engine like Google | [queue.acm.org](http://queue.acm.org/detail.cfm?id=988407)<br/>[stackexchange.com](http://programmers.stackexchange.com/questions/38324/interview-question-how-would-you-implement-google-search)<br/>[ardendertat.com](http://www.ardendertat.com/2012/01/11/implementing-search-engines/)<br>[stanford.edu](http://infolab.stanford.edu/~backrub/google.html) |
|
||||
| Design a scalable web crawler like Google | [quora.com](https://www.quora.com/How-can-I-build-a-web-crawler-from-scratch) |
|
||||
| Design Google docs | [code.google.com](https://code.google.com/p/google-mobwrite/)<br/>[neil.fraser.name](https://neil.fraser.name/writing/sync/) |
|
||||
| Design a key-value store like Redis | [slideshare.net](http://www.slideshare.net/dvirsky/introduction-to-redis) |
|
||||
| Design a cache system like Memcached | [slideshare.net](http://www.slideshare.net/oemebamo/introduction-to-memcached) |
|
||||
| Design a recommendation system like Amazon's | [hulu.com](http://tech.hulu.com/blog/2011/09/19/recommendation-system.html)<br/>[ijcai13.org](http://ijcai13.org/files/tutorial_slides/td3.pdf) |
|
||||
| Design a tinyurl system like Bitly | [n00tc0d3r.blogspot.com](http://n00tc0d3r.blogspot.com/) |
|
||||
| Design a chat app like WhatsApp | [highscalability.com](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html)
|
||||
| Design a picture sharing system like Instagram | [highscalability.com](http://highscalability.com/flickr-architecture)<br/>[highscalability.com](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html) |
|
||||
| Design the Facebook news feed function | [quora.com](http://www.quora.com/What-are-best-practices-for-building-something-like-a-News-Feed)<br/>[quora.com](http://www.quora.com/Activity-Streams/What-are-the-scaling-issues-to-keep-in-mind-while-developing-a-social-network-feed)<br/>[slideshare.net](http://www.slideshare.net/danmckinley/etsy-activity-feeds-architecture) |
|
||||
| Design the Facebook timeline function | [facebook.com](https://www.facebook.com/note.php?note_id=10150468255628920)<br/>[highscalability.com](http://highscalability.com/blog/2012/1/23/facebook-timeline-brought-to-you-by-the-power-of-denormaliza.html) |
|
||||
| Design the Facebook chat function | [erlang-factory.com](http://www.erlang-factory.com/upload/presentations/31/EugeneLetuchy-ErlangatFacebook.pdf)<br/>[facebook.com](https://www.facebook.com/note.php?note_id=14218138919&id=9445547199&index=0) |
|
||||
| Design a graph search function like Facebook's | [facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-building-out-the-infrastructure-for-graph-search/10151347573598920)<br/>[facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-indexing-and-ranking-in-graph-search/10151361720763920)<br/>[facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-the-natural-language-interface-of-graph-search/10151432733048920) |
|
||||
| Design a content delivery network like CloudFlare | [cmu.edu](http://repository.cmu.edu/cgi/viewcontent.cgi?article=2112&context=compsci) |
|
||||
| Design a trending topic system like Twitter's | [michael-noll.com](http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/)<br/>[snikolov .wordpress.com](http://snikolov.wordpress.com/2012/11/14/early-detection-of-twitter-trends/) |
|
||||
| Design a random ID generation system | [blog.twitter.com](https://blog.twitter.com/2010/announcing-snowflake)<br/>[github.com](https://github.com/twitter/snowflake/) |
|
||||
| Return the top k requests during a time interval | [ucsb.edu](https://icmi.cs.ucsb.edu/research/tech_reports/reports/2005-23.pdf)<br/>[wpi.edu](http://davis.wpi.edu/xmdv/docs/EDBT11-diyang.pdf) |
|
||||
| Design a system that serves data from multiple data centers | [highscalability.com](http://highscalability.com/blog/2009/8/24/how-google-serves-data-from-multiple-datacenters.html) |
|
||||
| Design an online multiplayer card game | [indieflashblog.com](http://www.indieflashblog.com/how-to-create-an-asynchronous-multiplayer-game.html)<br/>[buildnewgames.com](http://buildnewgames.com/real-time-multiplayer/) |
|
||||
| Design a garbage collection system | [stuffwithstuff.com](http://journal.stuffwithstuff.com/2013/12/08/babys-first-garbage-collector/)<br/>[washington.edu](http://courses.cs.washington.edu/courses/csep521/07wi/prj/rick.pdf) |
|
||||
| Add a system design question | [Contribute](#contributing) |
|
||||
|
||||
### Real world architectures
|
||||
|
||||
> Articles on how real world systems are designed.
|
||||
|
||||
<p align="center">
|
||||
<img src="http://i.imgur.com/TcUo2fw.png">
|
||||
<br/>
|
||||
<i><a href=https://www.infoq.com/presentations/Twitter-Timeline-Scalability>Source: Twitter timelines at scale</a></i>
|
||||
</p>
|
||||
|
||||
**Don't focus on nitty gritty details for the following articles, instead:**
|
||||
|
||||
* Identify shared principles, common technologies, and patterns within these articles
|
||||
* Study what problems are solved by each component, where it works, where it doesn't
|
||||
* Review the lessons learned
|
||||
|
||||
|Type | System | Reference(s) |
|
||||
|---|---|---|
|
||||
| Data processing | **MapReduce** - Distributed data processing from Google | [research.google.com](http://static.googleusercontent.com/media/research.google.com/zh-CN/us/archive/mapreduce-osdi04.pdf) |
|
||||
| Data processing | **Spark** - Distributed data processing from Databricks | [slideshare.net](http://www.slideshare.net/AGrishchenko/apache-spark-architecture) |
|
||||
| Data processing | **Storm** - Distributed data processing from Twitter | [slideshare.net](http://www.slideshare.net/previa/storm-16094009) |
|
||||
| | | |
|
||||
| Data store | **Bigtable** - Distributed column-oriented database from Google | [harvard.edu](http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/chang06bigtable.pdf) |
|
||||
| Data store | **HBase** - Open source implementation of Bigtable | [slideshare.net](http://www.slideshare.net/alexbaranau/intro-to-hbase) |
|
||||
| Data store | **Cassandra** - Distributed column-oriented database from Facebook | [slideshare.net](http://www.slideshare.net/planetcassandra/cassandra-introduction-features-30103666)
|
||||
| Data store | **DynamoDB** - Document-oriented database from Amazon | [harvard.edu](http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf) |
|
||||
| Data store | **MongoDB** - Document-oriented database | [slideshare.net](http://www.slideshare.net/mdirolf/introduction-to-mongodb) |
|
||||
| Data store | **Spanner** - Globally-distributed database from Google | [research.google.com](http://research.google.com/archive/spanner-osdi2012.pdf) |
|
||||
| Data store | **Memcached** - Distributed memory caching system | [slideshare.net](http://www.slideshare.net/oemebamo/introduction-to-memcached) |
|
||||
| Data store | **Redis** - Distributed memory caching system with persistence and value types | [slideshare.net](http://www.slideshare.net/dvirsky/introduction-to-redis) |
|
||||
| | | |
|
||||
| File system | **Google File System (GFS)** - Distributed file system | [research.google.com](http://static.googleusercontent.com/media/research.google.com/zh-CN/us/archive/gfs-sosp2003.pdf) |
|
||||
| File system | **Hadoop File System (HDFS)** - Open source implementation of GFS | [apache.org](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) |
|
||||
| | | |
|
||||
| Misc | **Chubby** - Lock service for loosely-coupled distributed systems from Google | [research.google.com](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/chubby-osdi06.pdf) |
|
||||
| Misc | **Dapper** - Distributed systems tracing infrastructure | [research.google.com](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36356.pdf)
|
||||
| Misc | **Kafka** - Pub/sub message queue from LinkedIn | [slideshare.net](http://www.slideshare.net/mumrah/kafka-talk-tri-hug) |
|
||||
| Misc | **Zookeeper** - Centralized infrastructure and services enabling synchronization | [slideshare.net](http://www.slideshare.net/sauravhaloi/introduction-to-apache-zookeeper) |
|
||||
| | Add an architecture | [Contribute](#contributing) |
|
||||
|
||||
### Company architectures
|
||||
|
||||
| Company | Reference(s) |
|
||||
|---|---|
|
||||
| Amazon | [Amazon architecture](http://highscalability.com/amazon-architecture) |
|
||||
| Cinchcast | [Producing 1,500 hours of audio every day](http://highscalability.com/blog/2012/7/16/cinchcast-architecture-producing-1500-hours-of-audio-every-d.html) |
|
||||
| DataSift | [Realtime datamining At 120,000 tweets per second](http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html) |
|
||||
| DropBox | [How we've scaled Dropbox](https://www.youtube.com/watch?v=PE4gwstWhmc) |
|
||||
| ESPN | [Operating At 100,000 duh nuh nuhs per second](http://highscalability.com/blog/2013/11/4/espns-architecture-at-scale-operating-at-100000-duh-nuh-nuhs.html) |
|
||||
| Google | [Google architecture](http://highscalability.com/google-architecture) |
|
||||
| Instagram | [14 million users, terabytes of photos](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html)<br/>[What powers Instagram](http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances) |
|
||||
| Justin.tv | [Justin.Tv's live video broadcasting architecture](http://highscalability.com/blog/2010/3/16/justintvs-live-video-broadcasting-architecture.html) |
|
||||
| Facebook | [Scaling memcached at Facebook](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/key-value/fb-memcached-nsdi-2013.pdf)<br/>[TAO: Facebook’s distributed data store for the social graph](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/data-store/tao-facebook-distributed-datastore-atc-2013.pdf)<br/>[Facebook’s photo storage](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf) |
|
||||
| Flickr | [Flickr architecture](http://highscalability.com/flickr-architecture) |
|
||||
| Mailbox | [From 0 to one million users in 6 weeks](http://highscalability.com/blog/2013/6/18/scaling-mailbox-from-0-to-one-million-users-in-6-weeks-and-1.html) |
|
||||
| Pinterest | [From 0 To 10s of billions of page views a month](http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html)<br/>[18 million visitors, 10x growth, 12 employees](http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html) |
|
||||
| Playfish | [50 million monthly users and growing](http://highscalability.com/blog/2010/9/21/playfishs-social-gaming-architecture-50-million-monthly-user.html) |
|
||||
| PlentyOfFish | [PlentyOfFish architecture](http://highscalability.com/plentyoffish-architecture) |
|
||||
| Salesforce | [How they handle 1.3 billion transactions a day](http://highscalability.com/blog/2013/9/23/salesforce-architecture-how-they-handle-13-billion-transacti.html) |
|
||||
| Stack Overflow | [Stack Overflow architecture](http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html) |
|
||||
| TripAdvisor | [40M visitors, 200M dynamic page views, 30TB data](http://highscalability.com/blog/2011/6/27/tripadvisor-architecture-40m-visitors-200m-dynamic-page-view.html) |
|
||||
| Tumblr | [15 billion page views a month](http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html) |
|
||||
| Twitter | [Making Twitter 10000 percent faster](http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster)<br/>[Storing 250 million tweets a day using MySQL](http://highscalability.com/blog/2011/12/19/how-twitter-stores-250-million-tweets-a-day-using-mysql.html)<br/>[150M active users, 300K QPS, a 22 MB/S firehose](http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html)<br/>[Timelines at scale](https://www.infoq.com/presentations/Twitter-Timeline-Scalability)<br/>[Big and small data at Twitter](https://www.youtube.com/watch?v=5cKTP36HVgI)<br/>[Operations at Twitter: scaling beyond 100 million users](https://www.youtube.com/watch?v=z8LU0Cj6BOU) |
|
||||
| Uber | [How Uber scales their real-time market platform](http://highscalability.com/blog/2015/9/14/how-uber-scales-their-real-time-market-platform.html) |
|
||||
| WhatsApp | [The WhatsApp architecture Facebook bought for $19 billion](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html) |
|
||||
| YouTube | [YouTube scalability](https://www.youtube.com/watch?v=w5WVu624fY8)<br/>[YouTube architecture](http://highscalability.com/youtube-architecture) |
|
||||
|
||||
### Company engineering blogs
|
||||
|
||||
> Architectures for companies you are interviewing with.
|
||||
>
|
||||
> Questions you encounter might be from the same domain.
|
||||
|
||||
* [Airbnb Engineering](http://nerds.airbnb.com/)
|
||||
* [Atlassian Developers](https://developer.atlassian.com/blog/)
|
||||
* [Autodesk Engineering](http://cloudengineering.autodesk.com/blog/)
|
||||
* [AWS Blog](https://aws.amazon.com/blogs/aws/)
|
||||
* [Bitly Engineering Blog](http://word.bitly.com/)
|
||||
* [Box Blogs](https://www.box.com/blog/engineering/)
|
||||
* [Cloudera Developer Blog](http://blog.cloudera.com/blog/)
|
||||
* [Dropbox Tech Blog](https://tech.dropbox.com/)
|
||||
* [Engineering at Quora](http://engineering.quora.com/)
|
||||
* [Ebay Tech Blog](http://www.ebaytechblog.com/)
|
||||
* [Evernote Tech Blog](https://blog.evernote.com/tech/)
|
||||
* [Etsy Code as Craft](http://codeascraft.com/)
|
||||
* [Facebook Engineering](https://www.facebook.com/Engineering)
|
||||
* [Flickr Code](http://code.flickr.net/)
|
||||
* [Foursquare Engineering Blog](http://engineering.foursquare.com/)
|
||||
* [GitHub Engineering Blog](http://githubengineering.com/)
|
||||
* [Google Research Blog](http://googleresearch.blogspot.com/)
|
||||
* [Groupon Engineering Blog](https://engineering.groupon.com/)
|
||||
* [Heroku Engineering Blog](https://engineering.heroku.com/)
|
||||
* [Hubspot Engineering Blog](http://product.hubspot.com/blog/topic/engineering)
|
||||
* [High Scalability](http://highscalability.com/)
|
||||
* [Instagram Engineering](http://instagram-engineering.tumblr.com/)
|
||||
* [Intel Software Blog](https://software.intel.com/en-us/blogs/)
|
||||
* [Jane Street Tech Blog](https://blogs.janestreet.com/category/ocaml/)
|
||||
* [LinkedIn Engineering](http://engineering.linkedin.com/blog)
|
||||
* [Microsoft Engineering](https://engineering.microsoft.com/)
|
||||
* [Microsoft Python Engineering](https://blogs.msdn.microsoft.com/pythonengineering/)
|
||||
* [Netflix Tech Blog](http://techblog.netflix.com/)
|
||||
* [Paypal Developer Blog](https://devblog.paypal.com/category/engineering/)
|
||||
* [Pinterest Engineering Blog](http://engineering.pinterest.com/)
|
||||
* [Quora Engineering](https://engineering.quora.com/)
|
||||
* [Reddit Blog](http://www.redditblog.com/)
|
||||
* [Salesforce Engineering Blog](https://developer.salesforce.com/blogs/engineering/)
|
||||
* [Slack Engineering Blog](https://slack.engineering/)
|
||||
* [Spotify Labs](https://labs.spotify.com/)
|
||||
* [Twilio Engineering Blog](http://www.twilio.com/engineering)
|
||||
* [Twitter Engineering](https://engineering.twitter.com/)
|
||||
* [Uber Engineering Blog](http://eng.uber.com/)
|
||||
* [Yahoo Engineering Blog](http://yahooeng.tumblr.com/)
|
||||
* [Yelp Engineering Blog](http://engineeringblog.yelp.com/)
|
||||
* [Zynga Engineering Blog](https://www.zynga.com/blogs/engineering)
|
||||
|
||||
#### Source(s) and further reading
|
||||
|
||||
* [kilimchoi/engineering-blogs](https://github.com/kilimchoi/engineering-blogs)
|
||||
|
||||
## Under development
|
||||
|
||||
Interested in adding a section or helping complete one in-progress? [Contribute](#contributing)!
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user