Creating an AES256-GCM Shellcode Crypter

By August 30, 2018 SLAE-x86

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:
Student ID: SLAE-1134
Assignment number: 7
Github repo:


Hey everyone! We’re going to attack the final SLAE exam component for the SecurityTube Linux Assembly Expert course. This one was a bit harder to get right, but is one that comes with some fantastic value. Here, we’re going to create a “custom” shellcode encrypter. I put custom into quotes because we’re using a well known encryption algorithm and libraries to do it – because writing your own encryption is sometimes more dangerous than not using encryption.


  • Create a custom crypter like the one shown in the “crypters” video
  • Free to use any existing encryption schema
  • Can use any programming language

Which encryption algorithm to use?

When I thought of a crypter, I immediately thought of AES256-GCM encryption. This encryption technique is one that is recommended by the United States National Security Agency, is considered very strong, and is resistant to tampering. Unlike other modes of AES encryption, AES-GCM encryption includes a second important step to the process. First, it performs encryption on the data using a key and an initialization vector (IV). After this though, you do not simply have the encrypted ciphertext, you also have a tag, which is the message authentication code (MAC) which can be used to then validate that the ciphertext message you received has not been tampered with. Thus, it allows us to keep a checksum with our encrypted ciphertext.

I felt this was an important type of crypter because in many cases our shellcode may be getting transformed or passed via a plaintext protocol such as HTTP, Email, etc. As such, in the same way that we can perform man in the middle attacks, so to can another malicious actor whom we want to protect ourselves against.

Writing our crypter


So with our encryption algorithm decided, we need to actually implement this. I chose to do this in C using two different files — one, the encrypt.c file which is used to take our shellcode and perform the encryption. This is shown below:

Please be sure to read the header in this file, to use this, you do need some packages that aren’t on Ubuntu by default. We can get these with:

sudo apt-get update && sudo apt-get install openssl libssl-dev

If you are comfortable with the files that will be installed, answer y when prompted to install the software. With that installed, the code then defines a generic (albeit poor quality) handle errors function just to let us know if something goes wrong. It shouldn’t, but it could. We then define the encryption function which leverages well known OpenSSL EVP (EVP is short for Digital EnVeloPe) encryption for AES-GCM. It returns the length of our ciphertext, another word for our encrypted data.

Finally, we reach our main function which is where the custom parts really are. In this, we define on line 76 the shellcode which we would like to encrypt. Since we’re using C, this should be null free as a null may terminate your string early! You’ve been warned!

We then define variables to hold our key and initialization vector. We prompt the user for the hostname of the remote machine, and then generate a key from this value. We generate a random IV, perform our encryption and then take care to print out all of the data we’re going to need in our decrypter.

During testing, it got frustrating to format it right, so this actually outputs the C code that you will need to paste into the decrypter.


Nice! We’re encrypting our shellcode now. But we don’t have a way to test it works. As such, we need to create a decryption and execution harness. This is our decrypt.c file. The key area of interest from a usage perspective are lines 85–88.

Here, similar to our encryption file, we have a generic error handler, then our decryption function. We’re not going to go into the specifics of what each line is doing, as there is better documentation available if you want to understand the math behind why it does what it does. There are comments though explaining the decryption process (or at least what we’re doing). The key is we’re returning a positive value if the decryption succeeded and a negative value if we encountered an issue such as the MAC didn’t match what we decrypted. This protects the binary against tampering!

Below that, we have our main function which is the magic of this. We have our output from the initial encryption process, we use gethostname to retrieve the hostname which will act as our decryption key. In transit, this means that even if it ends up getting executed somewhere else, it won’t work. This can also help us avoid antivirus which does more dynamic analysis, since it would need to be configured for the correct hostname to evaluate it.

We then generate our actual encryption key based off of the hostname (most hostnames aren’t long enough to be a strong encryption key), perform our decryption, and then we do something interesting which differs from many other crypters. We check what happened. Because of the MAC / tag, we are able to verify if the item we decrypted was modified in transit or by a foreign party. If we think it may have been modified, we exit without executing the decrypted shellcode, for our own safety. If we did successfully decrypt it though, we place a null terminator at the end of the decrypted text and execute our shellcode using a “standard” shellcode harness.

When we use the included compile scripts (or the compile commands listed in the file headers with our execve-stack shellcode, we see that it works perfectly!

successful encryption and decryption of our shellcode

Kevin Kirsche

Author Kevin Kirsche

Kevin is a Principal Security Architect with Verizon. He holds the OSCP, OSWP, OSCE, and SLAE certifications. He is interested in learning more about building exploits and advanced penetration testing concepts.

More posts by Kevin Kirsche

Leave a Reply