We found the encryptor but it won’t decrypt encrypted?

We intercepted some foreign documents. We think there’s interesting information inside but the file is protected with a unique password algorithm

Upload a 64-bit ELF shared object of size at most 1024 bytes. It should spawn a shell when loaded using LD_PRELOAD

We are provided with a server that implements a custom protocol and a packet capture of the protocol being used.

For this challenge we start off with a program which is aptly named broken as it segfaults when run.