Lately i have been working on something that required me to read text files having approx 200,00,000 lines using C.
Using the normal method( int a ) of declaring an integer array with 200,00,000 elements always ends up in a Segmentation Fault(Memory Overflow).
Answer to this problem was fairly trivial.
There are typically two kinds of memory associated with programs, Stack Memory and Heap Memory.
This type of memory stores all the local variables and the function calls made in a program.
Normal variable declarations like,
int a; int b;
happen in Stack Memory during compile time.
Stack Memory is generally small in size(usually less than 8MB).
Small size of the Stack Memory leads to memory overflow in situations where we try to use large amount of space(declare really large arrays or an array of a structure) or in recursive programs where the depth of recursion becomes really large( because function calls are also stored in the Stack Memory) and hence we get a Segmentation Fault.
Heap Memory on the other hand is used in case of dynamic allocations( mallocs ) like,
int *a = (int*)malloc(length*sizeof(int));
Size of the Heap Memory is only limited by the size of the RAM and the swap memory. Memory allocation happens at runtime.
This is like a free pool of memory for programs from which they can request some space for their use and then put it back in the pool( using free() ).
Heap memory is mostly used in situations where large amount of memory is needed(when Stack Memory is not enough).
Example comparing Stack and Heap Memory
Let us try to declare an integer array with 108 elements.
1) Normal Declaration.
It reports an error saying “array size is too large”. Because Stack Memory does not have enough space to allocate to this array.
2) Dynamic Allocation
int *a; a = (int*)malloc(100000000*sizeof(int));
This piece of code executes successfully without any errors. The array was allocated space in the Heap Memory( malloc ).
A good practice is to free up the used Heap Space so that eventually you dont run out of the Heap Space too.
int *a; a = (int*)malloc(100000000*sizeof(int)); free(a);
To wrap-up, Stack Memory stores local variables and function calls and is limited in space. Heap Memory stores the dynamically allocated data, used when large amount of memory is needed.