Scalability of web request handling:

That leads to the first golden rule for scalability: every server contains exactly the same codebase and does not store any user-related data, like sessions or profile pictures, on local disc or memory.

Sessions need to be stored in a centralized data store which is accessible to all your application servers. It can be an external database or an external persistent cache, like Redis.

After “outsourcing” your sessions and serving the same codebase from all your servers, you can now create an image file from one of these servers (AWS calls this AMI - Amazon Machine Image.)

Scalability of caches:

Always try to use cache such as Redis

The cache is good as long as you can make keys properly that you would not need even a database

Memcache can be used to cache direct; but more efficient approach would be redis.

Caches are difficult when you cache the result of query; each time table’s element changes, the cached element has to be dropped.

Instead, better to use cache objects

Scalability of asynchronism:

Just imagine the scalability of your website if the script would upload these pre-rendered HTML pages to AWS S3 or Cloudfront or another Content Delivery Network!

Your website would be super responsive and could handle millions of visitors per hour!

RabbitMQ is one of many systems which help to implement async processing. You could also use ActiveMQ or a simple Redis list.

The basic idea is to have a queue of tasks or jobs that a worker can process.

Scalability for database:

means to denormalize right from the beginning and include no more Joins in any database query

Joins will now need to be done in your application code.

Some ideas of objects to cache:

user sessions (never use the database!) fully rendered blog articles activity streams user<->friend relationships

Generally, you should aim for maximal throughput with acceptable latency.

Another way to look at performance vs scalability:

If you have a performance problem, your system is slow for a single user. If you have a scalability problem, your system is fast for a single user but slow under heavy load.

In a distributed computer system, you can only support two of the following guarantees:

Consistency - Every read receives the most recent write or an error Availability - Every request receives a response, without guarantee that it contains the most recent version of the information Partition Tolerance - The system continues to operate despite arbitrary partitioning due to network failures

CP - consistency and partition tolerance Waiting for a response from the partitioned node might result in a timeout error. CP is a good choice if your business needs require atomic reads and writes.

AP - availability and partition tolerance Responses return the most recent version of the data available on a node, which might not be the latest. Writes might take some time to propagate when the partition is resolved.

AP is a good choice if the business needs allow for eventual consistency or when the system needs to continue working despite external errors.

Consistency Issues:

Weak Consistency: After a write, reads may or may not see it. A best effort approach is taken.

Weak consistency works well in real time use cases such as VoIP, video chat, and realtime multiplayer games

For example, if you are on a phone call and lose reception for a few seconds, when you regain connection you do not hear what was spoken during connection loss.

Strong Consistency:

After a write, reads will see it. Data is replicated synchronously.

Strong consistency works well in systems that need transactions.

Eventual Consistency

After a write, reads will eventually see it (typically within milliseconds). Data is replicated asynchronously.

DNS and email

There are two main patterns to support high availability: fail-over and replication.

Fail Over

Active Passive

Heartbeat from passive to active; if active fails -> passive takes on ip of previously active one

Master slave failover

Depends whether the passive one needs to hot boot or cold boot to resume the processes.

Active Active

In active-active, both servers are managing traffic, spreading the load between them.

If the servers are public-facing, the DNS would need to know about the public IPs of both servers.

If the servers are internal-facing, application logic would need to know about both servers.

– More hardware and complex

– Potential loss of data if active fails before replicating the data to passive node.

Replication

Describing Availability:

Availability is often quantified by uptime (or downtime) as a percentage of time the service is available

99.9 % will have 8h downtime per year whereas 99.99% will have 52 mins downtime in a year(8.6s a day)

DNS:

DNS results can also be cached by your browser or OS for a certain period of time, determined by the TTL

NS record (name server) - Specifies the DNS servers for your domain/subdomain. MX record (mail exchange) - Specifies the mail servers for accepting messages. A record (address) - Points a name to an IP address. CNAME (canonical) - Points a name to another name or CNAME (example.com to www.example.com) or to an A record.

Some DNS services can route traffic through various methods:

Weighted round robin Prevent traffic from going to servers under maintenance Balance between varying cluster sizes A/B testing Latency-based Geolocation-based

CDNs:

A content delivery network (CDN) is a globally distributed network of proxy servers, serving content from locations closer to the user.

The site’s DNS resolution will tell clients which server to contact.

Serving content from CDNs can significantly improve performance in two ways:

Users receive content at data centers close to them Your servers do not have to serve requests that the CDN fulfills

Push CDNs:

Push CDNs receive new content whenever changes occur on your server.

You take full responsibility for providing content, uploading directly to the CDN and rewriting URLs to point to the CDN.

Content is uploaded only when it is new or changed, minimizing traffic, but maximizing storage.

Pull CDNs:

Pull CDNs grab new content from your server when the first user requests the content

This results in a slower request until the content is cached on the CDN.

A time-to-live (TTL) determines how long content is cached.

Sites with heavy traffic work well with pull CDNs, as traffic is spread out more evenly with only recently-requested content remaining on the CDN.

– CDNs require changing URLs for static content to point to the CDN.

– Costly

– Stale content if update happens before TTL

Load Balancer:

Pick a worker to forward the request. The worker will be chosen based on different types of algorithms.

Load balancers are effective at:

Preventing requests from going to unhealthy servers

Preventing overloading resources

Helping eliminate single points of failure

Can be implemented using hardware or software using a proxy known as HAProxy.

Additional benefits include:

SSL termination

encryption decryption from load balancer so ease for actual workers

Removes the need to install X.509 certificates on each server

Session persistence

Issue cookies and route a specific client’s requests to same instance if the web apps do not keep track of sessions

Load balancers can route traffic based on various metrics, including:

Random Least loaded Session/cookies Round robin or weighted round robin Layer 4 Layer 7

Layer 4 and 7 means the layers of Network protocol standards

Layer 4:

load balancers look at info at the transport layer to decide how to distribute requests.

Generally, this involves the source, destination IP addresses, and ports in the header, but not the contents of the packet.

Performs NAT

Layer 7:

Layer 7 load balancers look at the application layer to decide how to distribute requests.

terminates network traffic, reads the message, makes a load-balancing decision, then opens a connection to the selected server.

can direct video traffic to servers that host videos while directing more sensitive user billing traffic to security-hardened servers.

At the cost of flexibility, layer 4 load balancing requires less time and computing resources than Layer 7, although the performance impact can be minimal on modern commodity hardware.

– The load balancer can become a performance bottleneck if it does not have enough resources or if it is not configured properly.

– increased complexity

– A single load balancer is a single point of failure, configuring multiple load balancers further increases complexity.

Horizontal Scaling:

Scaling out using commodity machines is more cost efficient and results in higher availability than scaling up a single server on more expensive hardware, called Vertical Scaling.

Scaling horizontally introduces complexity and involves cloning servers: Servers should be stateless: they should not contain any user-related data like sessions or profile pictures

Sessions can be stored in a centralized data store such as a database (SQL, NoSQL) or a persistent cache (Redis, Memcached)

Downstream servers such as caches and databases need to handle more simultaneous connections as upstream servers scale out

Reverse Proxy:

web server that centralizes internal services and provides unified interfaces to the public.

Secure, Scalability and Flexibility, SSL termination, Compression, Caching, Static content serve

Load balancer vs reverse proxy:

Deploying a load balancer is useful when you have multiple servers. Often, load balancers route traffic to a set of servers serving the same function.

Reverse proxies can be useful even with just one web server or application server

Solutions such as NGINX and HAProxy can support both layer 7 reverse proxying and load balancing.

Separating out the web layer from the application layer (also known as platform layer) allows you to scale and configure both layers independently.

Adding a new API results in adding application servers without necessarily adding additional web servers.

Microservices : Each service runs a unique process and communicates through a well-defined, lightweight mechanism to serve a business goal.

Service Discovery

Systems such as Consul, Etcd, and Zookeeper can help services find each other by keeping track of registered names, addresses, and ports.

Health checks help verify service integrity and are often done using an HTTP endpoint.

Both Consul and Etcd have a built in key-value store that can be useful for storing config values and other shared data.

Functional Partitioning:

Federation (or functional partitioning) splits up databases by function.

For example, instead of a single, monolithic database, you could have three databases: forums, users, and products, resulting in less read and write traffic to each database and therefore less replication lag

– not effective if your schema requires huge functions or tables.

– need to update your application logic to determine which database to read and write.

– Joining data from two databases is more complex with a server link.

Sharding:

Sharding distributes data across different databases such that each database can only manage a subset of the data

sharding results in less read and write traffic, less replication, and more cache hits.

If one shard goes down, the other shards are still operational, although you’ll want to add some form of replication to avoid data loss.

Common ways to shard a table of users is either through the user’s last name initial or the user’s geographic location.

– update your application logic to work with shards,

–set of power users on a shard could result in increased load to that shard compared to others

Denormalization:

Increase performance at the cost of additional write in the database

Constraints to help different parts of db to sync; hard.

SQL Tuning:

It’s important to benchmark and profile to simulate and uncover bottlenecks.

Benchmark - Simulate high-load situations with tools such as ab.

Profile - Enable tools such as the slow query log to help track performance issues.

Good things to do:

CHAR instead of VARCHAR for fixed-length fields

TEXT for large blocks of text such as blog posts

INT for larger numbers up to 2^32 or 4 billion

DECIMAL for currency to avoid floating point representation errors.

Avoid storing large BLOBS, store the location of where to get the object instead.

Set the NOT NULL constraint where applicable to improve search performance.

Use good indices.

Avoid expensive joins: i.e. denormalize where needed

NosQL:

Data is denormalized, and joins are generally done in the application code.

Most NoSQL stores lack true ACID transactions and favor eventual consistency.

BASE chooses availability over consistency.

Basically available - the system guarantees availability.

Soft state - the state of the system may change over time, even without input.

Eventual consistency - the system will become consistent over a period of time, given that the system doesn’t receive input during that period.

Key Value Stores:

Key-value stores provide high performance and are often used for simple data models or for rapidly-changing data, such as an in-memory cache layer

Document Value stores:

Couch-db , mongo-db etc store all data as documents.

JSON, XML type storage of the data

Wide column store:

A wide column store’s basic unit of data is a column (name/value pair)

Column can be grouped in column families

Super column families further group column families.

BigTable, Cassandra etc. are examples of such db.

Graph Database: Node is a record

Use to store data having many to many relationships

Neo4j , FlockDB etc.

SQL vs NoSQL: Temporary store: NO Non relational data: no Store huge data: NO Strict schema: S

Caching: CDN Web server -> Reverse proxies, Varnish e.g. Database -> to boost db performance Client -> in browser, server side or distinct cache layer Application -> Memcache , Redis

Database query cache issues: Hard to delete cached result with complex query, need to delete on small change Object level cache: Cache of user session , fully rendered web pages, activity streams, user graph data

Cache-aside is also referred to as lazy loading. Only requested data is cached, which avoids filling up the cache with data that isn’t requested.

Issues of cache aside: Cache miss involves three steps to fetch data; data can become stale: solved by write through or TTL.

Refresh-ahead can result in reduced latency vs read-through if the cache can accurately predict which items are likely to be needed in the future.

Asynchronism:

Message queues: Message queues receive, hold, and deliver messages.

An application publishes a job to the queue, then notifies the user of job status.

A worker picks up the job from the queue, processes it, then signals the job is complete

For example, if posting a tweet, the tweet could be instantly posted to your timeline, but it could take some time before your tweet is actually delivered to all of your followers.

Redis is a simple msg broker, RabbitMQ, AmazonSQS are some examples.

Task Queues:

Receive task and related data , runs them and delivers results.

e.g. Celery python support and scheduling

Back pressure:

Once queues fill up client gets 503.

HTTP Requests: GET: reads resource ; cacheable

PUT: creates or replaces a resource; not cacheable

PATCH: partailly update resouce; cacheable if response contains freshness info

DELETE, POST

RPC:

a client causes a procedure to execute on a different address space, usually a remote server

Remote calls are usually slower and less reliable than local calls

e.g. Protobuf, Thrift , Avro

Consists of : Client program : params pushed to stack like local procedure call Client stub procedure: packs procedure id and args into request msg Client Comm module: os sends msg from client to server Server comm module Server stub procedure : calls server procedure matching procedure id and passes given args

RPC is focused on exposing behaviors.

RPCs are often used for performance reasons with internal communications, as you can hand-craft native calls to better fit your use cases.

– RPC clients become tightly coupled to the service implementation.

– A new API must be defined for every new operation or use case.

REST: Representational State transfer

The server provides a representation of resources and actions that can either manipulate or get a new representation of resources.

All communication must be stateless and cacheable.

Qualities of restful instance:

identify resources: use same uri regardless of action

verbs

status response in http

must be fully accessible in a browser

REST is focused on exposing data

Great for horizontal scaling and partitioning

Security:

Principle of least privilege

Parameterized queries to prevent SQL injection

1 TB means 1 trillion, 1GB means 1 billion, 1MB means one millions

Latency comparison:

L1 cache: 0.5ns L2 cache: 7ns Mutex lock/unlock: 25ns Main mem reference: 100 ns Send 1kb over 1GBps network: 10us SSD read speed: 1Gb/sec around; read 1mb from ssd: 1ms Disk seek: 10ms read 1mb from disk: 30ms read 1mb from 1gbps: 10ms