RC: W12 D5 — Reflecting on my 12-week batch at the Recurse Center

May 3, 2024

Today was my last day at the Recurse Center. I spent the day traveling back home instead of coding at the hub and this gave me some time to reflect on what happened in the past 12 weeks. So this post will be a bit different from the usual ones.

Doing RC was an absolute blast: I met amazing people, I got to work on things I love, I learned a ton, and I was in the safest work environment I have ever known. Today, after 12 weeks, I feel extremely sad to leave, but I am also excited to go back home and incorporate the learnings from this experience into the rest of my life, both as a person and as a programmer.

When starting RC, I wanted to take a break from a regular job to have time to understand much more deeply how things work under the hood (hence the name of that blog!). In particular, my main goal was to deepen my understanding of databases by reimplementing some from scratch. Another major goal of mine was to start a blog: that’s something I had been wanting to do for a long time, but I had always restrained myself from doing so because I thought that what I could talk about would never be “interesting enough”.

So… What did I end up doing in those 12 weeks? I spent the vast majority of my time working on 3 big projects linked to deepening my understanding of storage systems:

• I re-implemented Bitcask from scratch (it is the default storage engine in Riak). My project is in this github repo.
• I re-implemented RocksDB from scratch (it is a log-structured database engine). My project is in this github repo.
• I started re-implementing Kafka from scratch (it is a distributed event-streaming platform). I spent only two days on this so my github repo is still a work in progress.

In all of these projects, I imposed a major constraint upon myself: I wanted to understand everything, so I forbade myself to use any external library. As a consequence, I had to re-implement from first principles a bunch of basic mechanisms and data structures (like read-write locks, bloom filters and red-black trees). This obviously made my projects much slower than they could have been, but was immensely beneficial for my purpose: learning and deepening my understanding of systems.

To do my databases/Kafka projects, I proceeded as follows:

• I read books (“Designing Data-Intensive Applications” by Martin Kleppmann, “Database internals” by Alex Petrov) and some research papers (that of Bitcask, that on the evolution of RocksRB) to understand the main principles that guide their design;
• I spent a lot of time reading the source code of the original projects (Bitcask written in Erlang, RocksDB written in C++, Kafka written in Java) to understand how the core components are organized and to understand some specific implementation choices;
• I referred to some tutorials (mostly LSM in a week) and innumerable blog posts to break up the pieces in a meaningful order.

I spent the remaining of my time doing other things for fun:

• I participated in a book club on “Structure and Interpretation of Computer Programs” ( a.k.a “the Wizard book”) and solved the exercises of the first two chapters in MIT/Scheme (here are my solutions);
• I implemented basic authentication checks from scratch after p. b. explained those during a workshop;
• I gave a workshop on database internals to share what I had learned with other recursers (here are the slides and the repo containing the live-coded key-value store);
• I did over 30 pair-programming sessions with 20 other recursers, either on their projects or mine, and learned from them every time.

Out of curiosity, I had a look at how many lines of code that summed up to: all projects combined, I wrote over 14,000 lines of code. That averages to about 200 lines of code per day (as I was working most weekends on top of the 60 “regular” week days of my 12-week batch).

I also wrote one blog post every day to talk about one thing I learned that day. This was definitely a huge challenge, as I spent about 2-3 hours per day writing it. But I think it was incredibly valuable as this helped me:

1. Clarify my thought process: Writing down my thoughts helped me structure my reasoning and often made me realize that some bits that I thought I had understood were actually still blurry in my mind;
2. Force myself to really understand things: If I was planning to talk about a bug I encountered, merely solving it in a it-works-but-I-am-not-sure-why fashion was not enough. I needed to really understand what was happening from first principles to be able to explain it;
3. Stop worrying about what others would think if what I did was “not enough” or worse, “not good enough” or “not interesting enough”: my goal was to write one thing every single day, which meant that I could not afford to spend a whole day editing each post. I had to be okay with the fact that what I was sharing would not be perfect and rather focus on getting things done.

So that’s basically what I did: I worked on big projects to gain depth on systems which I originally had a superficial understanding of. This definitely worked great for me: I learned a lot on the topics I wanted to focus on. That being said, there are plenty of other ways to do RC. For example, some other recursers preferred doing many short projects instead of a few big ones, some decided to follow existing courses/books/tutorials to learn a new field, some enjoyed doing mostly pair-programming on other people’s projects without having a very defined project of their own, etc… The range of different topics that people focused on was also very broad: some did very creative/art-related work, some created web or mobile apps, some explored machine learning, some dived into low-level stuff and systems, some focused on security/crypto, some worked on audio synthesis, etc… Seeing the vastness of all those different fields and approaches that we all took was really inspiring: there is so much to learn and so many different ways to get there!

Aside from the programming work itself, the one thing I was probably the most amazed at was how safe the Recurse Center is and how incredibly helpful such an environment is to grow and learn: I had absolutely nothing to think/worry about and therefore, I was able to focus 100% of my time on programming, learning and growing. Working in this space taught me a lot as to how to create the best work environment possible and I am so grateful to the founders and facilitators for the amazing work they have done thinking about this matter and creating such a space!

All in all, doing RC was an incredible and transformative experience. If it sounds like something you would like to do, consider applying! As for me, it is time to enjoy some long-awaited holidays and take a small break from blogging: I will come back for more posts later this year!