Story: Redis and its creator antirez

Table of contents

This article is translated from the original Chinese edition.

In the world of databases, Redis stands out as unique. Instead of the usual tables or documents that are the central focus of most databases, with Redis, you interact directly with low-level data structures such as linked lists and hash tables.

This is all thanks to the innovative design of Redis creator Salvatore Sanfilippo, known online as antirez. A master in system programming, antirez prefers to think in terms of data structures like lists, hashes, and sets. The higher-level data representations, such as tables or documents, just didn’t suit his taste. When creating Redis, antirez was an amateur in the database field. But perhaps it was this lack of experience that allowed him to bring fresh ideas to the industry.

Today, Redis has become mainstream, with its footprint visible in nearly all web services. According to Stack Overflow’s surveys, Redis has been the most beloved database for five consecutive years.

antirez’s early life

Growing up in Sicily, Italy, antirez’s father was an electrician at a petrochemical company, where he dealt with industrial automation problems. In the 80s, his father started working with programmable logic controllers (PLCs), and his interest in these controllers led him to purchase a Z80 processor board and start programming on it.

When antirez was six, his family got their first personal computer, a TI99/4A. While it wasn’t commercially successful, it was an interesting machine. His father would write some BASIC programs on it, and antirez quickly followed suit, learning to type and copy BASIC programs by the time he was in first grade.

As he grew older, antirez continued programming. However, by the age of 14, typical teenage interests like motorcycles and girls began to take over, and programming took a back seat. It wasn’t until he was 18 or 19 that he rediscovered his passion for computers, playing with 3D modeling and games, and writing simple programs. While studying architecture at the University of Palermo, he used BASIC for drawing, the only programming language he knew from his childhood.

SECLAB

One day, antirez wanted to buy a graphics card but mistakenly purchased a modem instead. When the retailer refused a refund, he just went with it, saying, “Well, I guess it’s a modem then!” It was the 90s, and the internet was becoming a thing. He got online, installed Linux, and began taking an interest in information security. He bought a book on the C programming language and started learning. Before long, he discovered a vulnerability in the ping program.

The vulnerability was this: Unix typically has a restriction that if you’re not root, you can’t quickly send a large number of packets in a short period. But antirez found a way to bypass this restriction using Unix signals.

He posted his findings on the Bugtraq mailing list in April 1998. Bugtraq was a mailing list about cybersecurity, where security experts from around the world would publish security issues. The next day, antirez got a long-distance call from Milan; a cybersecurity company named SECLAB wanted him to work for them.

Surprised, antirez responded, “But I’m just an architecture student. I don’t know anything.” But the owner of SECLAB still encouraged him to come to Milan for a chat. After meeting with them, the owner told antirez, “Go on with your life, but read these ten books and then come back to me.” The books were all about networking, like TCP/IP protocols, firewalls, application security, and cryptography. So, in the summer of 1998, antirez spent his days on the shores of Sicily reading these books.

In September, antirez returned to SECLAB, who wanted to hire him. He took a break from university and moved to Milan for work. During his first meeting, he couldn’t understand a thing his colleagues were saying, and he felt like a complete impostor. Some of Italy’s best hackers worked at SECLAB at the time. antirez began to contribute after two months of immersion.

First, antirez invented an attack called Idle Scan, which is still one of the famous attacks in Nmap and is still being studied. The owner of SECLAB saw more value in antirez’s contributions to the open-source community than in dealing with clients, so he was encouraged to focus on writing open-source software and researching cyber attacks.

Additionally, antirez created a tool called hping, a command-line tool that lets you send custom TCP/IP packets. It served a similar purpose to ping but didn’t have to follow the ICMP protocol like ping, so hping could do more, such as testing firewall rules, testing network performance, and transmitting files past firewalls.

After six months at SECLAB, antirez left. He had begun to stay up late and arrive late to work. Milan, a bustling city, was a bit too tempting for a 21-year-old, and it was easy to lose oneself without good self-discipline. Even though his time there was brief, antirez wrote a lot of C programs and participated in many projects during those six months, marking a significant turning point in his career.

Starting Merzia, his own company

In 2000, antirez joined Linuxcare Italia, a company dealing with open-source software. He was able to work remotely from home, and everything was going great until the dot-com bubble burst. One day, the company sent an email to its employees, essentially saying, “We’re shutting down operations in 18 countries, and in a month, you’re all going to be out of a job. Good night.”

In 2001, at the age of 23 and with a child to support, antirez needed to find a way to make ends meet. He began writing PHP, taking care of both front-end and back-end development, and working on small projects for clients.

2005 was the dawn of Web 2.0, and he and his friend Fabio Pitrola founded a company named Merzia, aiming to create Web 2.0 applications for the Italian market. Telecom Italia, the major Italian telecommunication company, saw what they were doing and reached out to establish a collaborative relationship. Telecom Italia eventually bought two websites they developed: OKNOtizie, a social news site, and Segnalo, a bookmarking website similar to Delicious.

The birth of LLOOGG and Redis

A few years later, antirez and his friend decided to focus on a new field: web analytics. They aimed to create a tool for bloggers that would allow them to monitor their visitors’ real-time behavior. For example, a visitor clicks on an article from Google, returns to the homepage, and then navigates to a specific page. This information could help bloggers improve the design and navigation of their websites. This tool was called LLOOGG, and its home page contained a few lines that highlighted how it differed from Google Analytics.

What set LLOOGG apart was its ability to let website administrators track a visitor’s real-time activity on their site. Google Analytics didn’t introduce this feature until 2011, yet LLOOGG was launched in 2007, demonstrating its innovation.

At first, antirez tried to use MySQL, but soon ran into performance issues. Every time MySQL read or wrote data, it had to access the hard drive. As the data volume grew, operations became bottlenecked by hard drive read/write speeds, causing the website to grind to a halt. From a modern perspective, an experienced backend engineer would tell you not to use MySQL for real-time applications, but this was not so obvious back in 2007.

antirez believed that using memory could solve this issue, so he wrote a prototype of a memory database in the Tcl programming language. Named LMDB (LLOOGG Memory Database), this was the precursor to Redis. The source code was just over 300 lines long, but it was enough to give you a good idea of what Redis would come to be.

Firstly, the prototype already supported basic commands like SET, GET, LPUSH, RPUSH, and so on. Two data types were supported: strings and lists. Moreover, the communication protocol between the server and the client was the same one used by Redis v1. It was designed to be human-readable and fast to parse.

Redis’ default port number, 6379, was in the Tcl source code too. It has an interesting backstory. On a phone keyboard, 6379 is MERZ. It’s a term inspired by Alessia Merz, an Italian showgirl. Alessia’s amusing comments on TV often entertained antirez and his friends. Among friends, they would describe something silly or meaningless by saying “that’s merz!” Over a decade or so, the term “merz” gradually evolved among their circle to describe something that seemed foolish but had technical hack value. This probably also explains the origin of their company name, Merzia.

In the first few lines of the Tcl source code, antirez imported a package but didn’t use it, leaving only a comment, # For [fork]. He wanted to use the fork() system call for data persistence, a mechanism to write data back to the hard drive. This is still how Redis RDB works to this day: fork() creates a child process that writes data to the disk, while the parent process continues with its original tasks. The Tcl source code didn’t really use fork(), antirez just left a comment there. That was the time he realized that Tcl had reached its limits and it was time to rewrite it in C.

In February 2009, antirez’s friend David Welton helped him share Redis to the world on Hacker News. The response was pretty muted; apart from David, only four people responded. Three of them said there were already similar projects out there, and only one person responded positively and offered to help. That person was Ezra Zygmuntowicz (who was unfortunately deceased in 2014 RIP), who was already quite famous in the Ruby on Rails community and was a co-founder of EngineYard. Ezra wrote the initial version of the Ruby client for Redis, redis-rb, which remains the go-to Ruby client to this day. And redis-rb played a key role in making Redis popular in the Ruby community.

Over five years, Redis handled two billion page views for LLOOGG. When the site was shut down in 2014, Redis was processing 350-400 commands per second. And all of this was running on a virtual machine that cost just $150 a month.

Early adoption of GitHub, Instagram, and Twitter

Despite the muted response on Hacker News, antirez continued to develop Redis outside of his day job. This went on for about a year, gradually attracting users.

At the time, he had two other open-source projects: hping and the Jim Interpreter, but Redis was the first one he felt was worth long-term investment. After losing interest in cybersecurity, he stopped developing hping. Disillusioned with the direction of Tcl and embracing Ruby, he stopped working on the Jim Interpreter. But he was sure he couldn’t live without databases, so he assumed Redis would be a success and continued its development. Even if it didn’t succeed, he knew he’d be using it for years to come.

In 2009, the same year Redis was published, Chris Wanstrath, the CEO of a startup called GitHub, used Redis to build a job queue system named Resque for running GitHub’s background tasks. In the Rails world, Resque was the most popular job queue system at the time. A successor, Sidekiq, which emerged in 2012 and was also built on Redis, is now the top choice for Rails application developers.

Another startup, Instagram, began to get in touch with antirez in 2010. In those emails, one of its co-founders, Mike Krieger, discussed how to use Redis to build Instagram. Both Instagram and Redis were just getting started, and Mike and antirez didn’t know each other. Instagram was entirely built on Redis in its early years, so without Redis, Instagram might not have existed, or at least would have been delayed.

In 2010, Twitter also joined the ranks of Redis users. Twitter invited antirez to Twitter HQ several times to work on their new Redis-based solution for the timeline page. Interestingly, not long after Redis was launched (in March 2009), antirez built a Twitter clone called Retwis to showcase a Redis application.

VMware and Redis Labs sponsorship

For about a year, Redis was nothing more than a hobby project for antirez. He improved it and released new versions without being paid, until one day he received a call from VMware.

“We’d like to financially support Redis,” they said. “We have an office in Italy and we’d like to hire you. Would you like to be our employee?” When antirez asked what he would need to do for them, they reassured him, “No, no, no, you just continue what you’re currently doing and we’ll pay you. All you need to do is mention on your website that Redis is sponsored by VMware.”

So just like that, Redis got its first major sponsor. antirez worked with VMware on Redis for several years. When VMware later split with Pivotal, he moved to Pivotal.

As Redis gained more users, the first wave of companies offering Redis consulting services emerged. Among them was a company from Tel Aviv, Israel, called Redis Labs. Originally named Garantia Data and started as a database supplier, the company later renamed itself to Redis Labs and shifted its focus to Redis-related business.

Pivotal and Redis Labs shared a common investor who one day suggested to antirez, “What are you doing at VMware? Go to Redis Labs. That’s where your product’s support center is.” Consequently, antirez moved to Redis Labs, where he has been working ever since. In August 2021, Redis Labs officially changed its name to Redis, turning the project’s website, redis.io, into the company’s website.

Ten years later

Ten years after Redis was born, in January 2020, antirez stood on the stage at a Redis Day conference, introducing the new features of Redis 6.0 with his signature Italian-accented English. On Twitter, you can find some pictures of attendees with antirez on that day.

Look back on the growth of Redis over these ten years:

Time	Version	New features
Feb 2009	？	Strings, lists
Sep 2009	1.0	Sets, sorted sets, persistence (RDB, AOF), …
Sep 2010	2.0	Hashes, pub/sub, bitmaps, Lua scripting, HyperLogLog, …
Apr 2015	3.0	Clusters, geospatial indexes
Jul 2017	4.0	Modules: RediSearch, RedisJSON, RedisGraph, …
Oct 2018	5.0	Streams
Jan 2020	6.0	SSL、ACL, …

In June of the same year, antirez wrote a blog post saying that he had decided to step down from his role as the developer and maintainer of the Redis project. Although he would continue to serve as a consultant for Redis Labs, offering advice, he would no longer be directly involved in development. Over a decade of devotion to the same project had left antirez feeling burned out.

For antirez, programming was a way to express himself, a form of art. Every character and line break had to be meticulously crafted, akin to the art form of writing. Software development was like writing a book — it had to be beautiful, elegant, and easy to comprehend. If that software happened to be useful to others, that was just a side effect.

Of course, antirez understood that this idea wasn’t practical. When your software is adopted on a large scale, you have to make many compromises that might render your software less than perfect, but more useful for people. Maintaining Redis became increasingly challenging, leaving him torn between art and utility. This struggle left antirez feeling exhausted and under pressure, yearning to return to his identity as an artist and create pure programming art.

After leaving the Redis project, antirez wrote a science fiction about artificial intelligence called Wohpe, which was published in July 2022. In April of the same year, the Redis team released version 7.0. And that was the first major version since antirez left the project.

References

Epilogue

This article went to the front page on Hacker News, and antirez pointed out some gaps and errors in the story. So I’ve made some updates to the article:

antirez was not fired in Milan. He quit himself to return to Sicily.
His first computer was TI99/4A instead of Texas Instruments I94A.
I added some details about how he was invited to Twitter HQ.

Be sure to check out the thread on Hacker News for more interesting discussions.