Reverse engineering or back engineering is a process to deconstruct software to analyze its internal working, design, and architecture. It is often used by security professionals to analyze the working of any malware or virus or software testing team as a black box testing to identify any security flaws.

The process of reverse engineering varies depending on the case presented, but a very simple approach would be:

• Information gathering
• Decompilation or debugging
• Identification of relevant information
• Report generation

This tutorial requires basic understanding of assembly language, take a crash course here

Information gathering

Information gather or better known as the behavioral analysis is the very first step a reverser should follow before attempting anything else. This includes gathering an idea of how the program works or how it should be working to gain or understand the outer working or getting an overview of the area you are interested in.

For example, let’s take a program that requires us to find out the valid keys in-order for us to use it.

Interesting! This application asks for the name and serial key, and does some internal check and based on that it will show either a failed message or maybe a success message. This information is sufficient for us (for this tutorial)

Decompilation and debugging

Now we have some useful information in our hand, let’s dive into the debugging first. I will be using x64 Dbg for the debugging part and IDA pro for this tutorial.

Fire up the x64dbg and load the application, once the application is loaded and running we need to search for our clues. Let’s start by searching for string Enter your name and bingo, we found it

Double-clicking on the string will take you to the code where the string is used, let’s have a look

Now if you take a closer look at the assembly code, you can figure out that:

• The application is using printf and scanf functions to print the message for name and serial number and then stores them
• Then the application is calling a function, which returns some value, let’s say X
• This value of X is then compared with the serial number which we provided and if they match it will show us the success message else, it will show failure message

So, what’s inside this function which returns the value of X? Let’s put a breakpoint on the function call and run the application and provide some dummy information. When the debugger hits the breakpoint, let’s step into this function.

By quickly analyzing the function, we can get an overview:

• This function takes a name as input
• Gets the length of name by calling strlen function
• Does some calculation on every character of the name, like by using while loop (take a look at JBE instruction)
• And finally formats the calculated value as an integer with %d-%d

Identification of relevant information

Being, a lazy person I am not going to run each step and analyze what’s exactly going on. If you wish, you can do it yourself, it’s easy assembly code and can be analyzed very easily.

So, here comes the role of IDA Pro. Note down the starting address of the function, then fire up the IDA Pro and load the application. Once the application is loaded, go to the address which you’ve noted down. By default, IDA will show you the graph view of that function

Now, select/highlight the function and press F5 key, and IDA will convert this assembly code to rough C code, which you can understand better

Report generation

Yeah, now we are getting close to unlocking this application. Let’s clean this source code which is generated by IDA. When cleaning is done, the function will look something like:

1
2
3
4
5
6
7
8
9
10
11
12
void generateSerial(char name[]) {
    int length = strlen(name);
    int counter = 1;
    int value = 0;
    while(counter <= length) {
        int index = counter - 1;
        int temp = value + counter  + 3;
        counter++;
        value = (unsigned int) (name[index] + temp);
    }
    sprintf(name, "%d-%d", 3 * (unsigned int)(length >> 1) + (unsigned int) length, value);
}

Now let’s generate serial for my name 😎

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#include <stdio.h>
#include <string.h>

#define MAX_NAME_LEN 20

void generateSerial(char name[]) {
    int length = strlen(name);
    int counter = 1;
    int value = 0;
    while(counter <= length) {
        int index = counter - 1;
        int temp = value + counter  + 3;
        counter++;
        value = (unsigned int) (name[index] + temp);
    }
    sprintf(name, "%d-%d", 3 * (unsigned int)(length >> 1) + (unsigned int) length, value);
}

int main() {
    char name[MAX_NAME_LEN];
    printf("Enter your name: ");
    fgets(name, sizeof name, stdin);
    name[strcspn(name, "\n")] = '\0';

    // calculate serial for the given name
    generateSerial(name);
    printf("Serial: %s\n", name);
    return 0;
}

For name, tejashwi our program will return key 20-923. Let’s run and verify!

You can download the sample application and keygen source code here and the password for the file is tejashwi.io

So this is an example of very very basic reverse engineering, if you are interested or thinking to dive into this, you should start by following the steps given below:

• Complete the fundamentals of C programming language
• Learn the basic of assembly programming language
• Learn about Windows PE file format
• And there’s a very famous video tutorial on windows reverse engineering made by Lena151, you can search on Google and find that. But let me help you, it’s here

And read this awesome blog https://lifeinhex.com/tag/lena151/

Till then, keep exploring