The purpose of the manifest file is to allow to start an engine from an existing set of SSTable files already written to disk. The files by themselves are not enough: we need to know which SSTable corresponds to which level and within each level, what is the correct ordering of SSTables. Getting this right is critical as it determines the ordering of records and thus which are served when a query is made on the engine. These pieces of information (correct ordering and levels of the SSTables) is stored in the state of the LSM engine. In my implementation, what I store in memory is a linked list of SSTables for each level.

The manifest records all the events that changed the state of the engine (i.e. all the events that ended in creating or deleting SSTables). Thus, by replaying the events, it is possible to recreate the correct state.

In practice, to recreate the full state, I need to:

1. Initialize a new LSM engine with the correct configuration (number of levels, max block and SSTable sizes, thresholds to trigger compaction, etc…);
2. Assign each relevant SSTable file to the level it belongs to after all the events have been replayed.

I stored the full configuration of the LSM engine in the header of the manifest. So parsing this chunk allows to initialize the engine easily.

Once the header has been parsed, the remaining content of the manifest corresponds to events. To extract the full list of events, I proceeded by:

• Reading and decoding the first event in the remaining stream of bytes;
• Moving the pointer to the end of that chunk;
• Repeating these two steps until the file has been read completely.

Here is how decoding the manifest file to extract the header and events can be done in practice:

class ManifestFile:
    def decode(self) -> tuple[ManifestHeader, list[Events]]:
        with open(self.path, "rb") as f:
            data = f.read()

        # Decode header
        header = ManifestHeader.from_bytes(data=data)
        checkpoint = header.size

        # Decode events
        events = self.decode_events(data=data[checkpoint:])

        return header, events

    @staticmethod
    def decode_events(data: bytes) -> list[Event]:
        events = []
        while len(data):
            manifest_record = ManifestRecord.from_bytes(data=data)
            event, checkpoint = manifest_record.event, manifest_record.size
            events.append(event)
            data = data[checkpoint:]
        return events

From this point on, recreating the full LSM engine becomes easy. First, it can be initialized with the configuration read from the header. Second, the events can be replayed to assign the SSTables to their correct level and thus recreate the state of the engine. The last thing that I will need to tackle is to actually create the manifest and write the events to it as they occur, but that will be for another day!