QNimate

  • CoursesVideos
  • WP PremiumPlugins
  • DemosLab
  • Home
  • QIdea
  • QTrack
Home Carbon Ads Overview Of Redis Architecture

Overview Of Redis Architecture

redis

Redis is a in-memory, key-value data store. Redis is the most popular key-value data store. Redis is used by all big IT brands in this world. Amazon Elastic Cache supports Redis which makes redis a very powerful and must know key-value data store. In this post I will provide you a brief introduction to redis architecture.


What Is In-Memory, Key-Value Store

Key-Value store is a storage system where data is stored in form of key and value pairs. When we say in-memory key-value store, by that we mean that the key-value pairs are stored in primary memory(RAM). So we can say that Redis stored data in RAM in form of key-value pairs.

In Redis, key has to be a string but value can be a string, list, set, sorted set or hash.

These are some example of Redis key-values pairs

name="narayan"
profession=["web", "mobile"]

Here name and profession are keys. And we have their respective values on right.

Advantage And Disadvantage of Redis over DBMS

Database Management systems store everything in second storage which makes read and write operations very slow. But Redis stores everything in primary memory which is very fast in read and write of data.

Primary memory is limited(much lesser size and expensive than secondary) therefore Redis cannot store large files or binary data. It can only store those small textual information which needs to be accessed, modified and inserted at a very fast rate. If we try to write more data than available memory then we will receive errors.

Redis Single Instance Architecture

Redis architecture contains two main processes: Redis client and Redis Server.
redis-client-server

Redis client and server can be in the same computer or in two different computers.

Redis server is responsible for storing data in memory. It handles all kinds of management and forms the major part of architecture. Redis client can be Redis console client or any other programming language’s Redis API.

As we saw that Redis stores everything in primary memory. Primary memory is volatile and therefore we will loose all stored data once we restart our Redis server or computer. Therefore we need a way for datastore persistance.

Redis Persistance

There are three different ways to make Redis persistance: RDB, AOF and SAVE command.

RDB Mechanism

RDB makes a copy of all the data in memory and stores them in secondary storage(permanent storage). This happens in a specified interval. So there is chance that you will loose data that are set after RDB’s last snapshot.

AOF

AOF logs all the write operations received by the server. Therefore everything is persistance. The problem with using AOF is that it writes to disk for every operation and it is a expensive task and also size of AOF file is large than RDB file.

SAVE Command

You can force redis server to create a RDB snapshot anytime using the redis console client SAVE command.

You can use AOF and RDB together to get best persistance result.

For more information of Redis persistance refer this official documentation.

Backup And Recovery Of Redis DataStore

Redis does not provide any mechanism for datastore backup and recovery. Therefore if there is any hard disk crash or any other kind of disaster then all data will be lost. You need to use some third party server backup and recovery softwares to workaround it.

If you are using Redis in a replicated environment then there is no need for backup.

Redis Replication

Replication is a technique involving many computers to enable fault-tolerance and data accessibility. In a replication environment many computers share the same data with each other so that even if few computers go down, all the data will be available.

This image shows a basic Redis replication
redis-replication

Master and slaves are redis servers configured as such.

All the slaves contain exactly same data as master. There can be as many as slaves per master server. When a new slave is inserted to the environment, the master automatically syncs all data to the slave.

All the queries are redirected to master server, master server then executes the operations. When a write operation occurs, master replicates the newly written data to all slaves. When a large number sort or read operation are made, master distributes them to the slaves so that a large number of read and sort operations can be executed at a time.

If a slave fails, then also the environment continues working. when the slave again starts working, the master sends updated data to the slave.

If there is a crash in master server and it looses all data then you should convert a slave to master instead of bringing a new computer as a master. If we make a new computer as master then all data in the environemt will be lost because new master will have no data and will makes the slaves also to have zero data(new master does resync ). If master fails but data is persistent(disk not crashed) then starting up the same master server again will bring up the whole environment to running mode.

Replication helped us from disk failures and other kinds of hardware failures. It also helped to execute multiple read/sort queries at a time.

For more detailed information on redis replication read this official documentation.

Persistance In Redis Replication

We saw how persistance can tackle unexpected failures and keep our backend strong. But one way whole data can be lost is if the whole replicated environment goes down due to power failure. This happens because all data is stored in primary memory. So we need persistance here also.

We can configure master or any one slave to store data in secondary storage using any method(AOF and RDB). Now when the whole environment is again started, make the persistent server as the master server.

Using persistance and replication together all our data is completely safe and protected from unexpected failures.

Clustering In Redis

Clustering is a technique by which data can be sharded(divided) into many computers. The main advantage is that more data can be stored in a cluster because its a combination of computers.

Suppose we have one redis server with 64GB of memory i.e., we can have only 64GB of data. Now if we have 10 clustered computers with each 64GB of RAM then we can store 640GB of data.

redis-cluster

In the above image we can see that data is sharded into four nodes. Each node is a redis server configured as a cluster node.

If one node fails then the whole cluster stops working.

More detailed information on redis cluster read this official documentation.

Persistance In Cluster

Data is stored in primary memory of nodes. We need to make the data of each node persistance. We can do that using the previously mentioned methods(AOF and RDF). Just configure every node for persistance storage.

Clustering And Replication Together

Suppose due to disk crash, one of our node goes down then the whole cluster stops working and never resumes. There is no way we can recover back the node as the data is completely lost.

To avoid this situation we can take a manual backup of each node regularly. But thats a tough and improper task. Therefore we can rely on replication to solve this problem.

cluster-replication-redis

Here we convert each node server to a master server. And we keep a slave for every master. So if any node(master) fails, the cluster will start using the slave to keep the cluster operating.

Redis Client

If you are first time getting yourself into redis then these links will be very helpful to install redis and learn redis client.

  1. Try Redis: This is a awesome online Redis console client which will help you to learn how to use Redis console client.
  2. Redis Quick Start: This article will help you to install redis and get started with it.
  3. FAQ’s: You can see the frequently asked questions about redis on this link.

Conclusion

Before writing this article I was trying to find articles on redis architecture. But I didn’t find any. Therefore I wrote a article about it. I tried to make it very simple so that anyone who is completely new to redis and DBMS can understand it. I hope I made everything clear. Leave comments for questions on it. Thanks for reading.

May 27, 2014Narayan Prusty
Creating A Simple Backdrop EffectSidebar for Phonegap App
Comments: 15
  1. rajesh
    4 years ago

    Thanks for nice and simple explanation.
    I am planning to deploy the same master and cluster architecture on raspberry pi 3 boards.
    With 1GB ram and 16GB memory, is it a good idea?
    Please suggest.

    ReplyCancel
  2. SHOUBHIK
    5 years ago

    Thank You Sir… For This Awesome Post… I Will Love To Gain More Knowledge From You… Thanx Again

    ReplyCancel
  3. Rishi
    5 years ago

    Hey Narayan,

    Nice article, nice way of explaining the whole process in one.
    However while reading this my mind took a tour of conventional ways to access data which was stored in secondary memory, no doubt Redis is helping us fetching data faster, here are few points which i thought can be valid at some point of time.
    1. Every company cannot afford this system, since its little bit expensive and costs servers ( master-slaves ).
    2. RAM usage on servers, it has to be fast and big to store the data, anyways if we restarts everything is gone and if we do not use the replication, with that said, Redis is somehow giving hint to use replication process which is again forcing you to have more servers.
    3. “If one node fails then the whole cluster stops working.” this line is little bit worrying, i know there are methods which will eventually help me to get rid of this.

    Dont we think its little expensive over a infrastructure cost and complex too.

    Your thoughts are welcome !

    ReplyCancel
  4. Rahul
    5 years ago

    Hi Narayan,

    This is the best article on Redis i could find on the internet, thanks a lot for putting your time on this.

    ReplyCancel
  5. Cong
    5 years ago

    master sends writes to slave asynchronously. This means when master instance dies. All new writes are gone. This an issue in Redis.

    ReplyCancel
  6. Colm
    5 years ago

    Thanks for putting in the time, makes it a lot clearer.

    Go raibh maith agat.

    Colm

    ReplyCancel
  7. ahzan
    6 years ago

    Awesome !!! article simply explained and precisely briefed , we expect more articles from you

    ReplyCancel
  8. Sirajudeen
    7 years ago

    This is my first choice when i started to learn about redis. Thanks a lot

    ReplyCancel
  9. Cheney
    7 years ago

    Hi NARAYAN,

    Thank you for the overview, this is the first article when I searched redis architecture, and it’s quite helpful.

    I’m a begginer of redis, and I have two questions here:

    Under “Redis Replication” section, you mentioned “If there is a crash in master server and it looses all data then you should convert a slave to master instead of bringing a new computer as a master”. Is this done manually?
    Looks to me this is a typical zookeeper user case, so the question is, is zookeeper node management solution already part of redis?
    I’m quite interested in redis cluster.
    Furthermore, I’m wondering if redis already support synchronization across different clusters (which can be different colos, I mean clusters in different data centers). For instance, I some <key, value> is set at east coast, is it possible to be available from west coast soon.

    Not sure if it’s a good place to post the questions here, appreciate any kind of guidance.

    ReplyCancel
    • Narayan Prusty
      7 years ago

      To make it automatic you need to use Sentinel.

      Regarding synchronization, yes it possible to do that. It all about IP address used while setting up replication.

      ReplyCancel
    • Maryland
      6 years ago

      I’m impressed! You’ve managed the almost imboesipls.

      ReplyCancel
  10. Saeid Zebardast
    7 years ago

    Thank you so much for great explanation.

    ReplyCancel
  11. Itamar Haber
    8 years ago

    Very nicely done. You may want to consider adding information about Sentinel – Redis’ mechanism for high-availability.

    ReplyCancel
    • pnarayanprusty
      8 years ago

      I wrote this for beginners, therefore didn’t include anything about sentinel. I will surely write one article about sentinel.

      ReplyCancel
      • Itamar Haber
        8 years ago

        Cool – perfectly acceptable :) While on the topic of Redis beginners, would it be too much to mention that the company I work at – http://redislabs.com – also offers a fully- managed Redis service? Redis Cloud is especially suited for developers that wish to concentrate on developing their applications instead of worrying about operating their Redis, sharding it for scalability, implementing HA mechanisms and tuning it for performance.

        ReplyCancel

Leave a Reply to Rishi Cancel reply

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax

Narayan Prusty

I am a software engineer specialising in Blockchain, DevOps and Go/JavaScript. This is my personal blog where I write about things that I learn and feel interesting to share.

Image8 years ago 21 Comments Other
Share this
0
GooglePlus
0
Facebook
0
Twitter
0
Linkedin
  • What Is In-Memory, Key-Value Store
  • Advantage And Disadvantage of Redis over DBMS
  • Redis Single Instance Architecture
  • Redis Persistance
  • Backup And Recovery Of Redis DataStore
  • Redis Replication
  • Persistance In Redis Replication
  • Clustering In Redis
  • Persistance In Cluster
  • Clustering And Replication Together
  • Redis Client
Related Articles
  • Storing Binary Data in Redis
  • Redis Permanent Storage
  • Streaming File Uploads to Storage Server with Node.js
  • How Does WordPress Cache Data?
  • Working with File System using Intel XDK
Our Sponsor
My Books

2014 - 2015 © QNimate
All tutorials MIT license