Hi Readers,
This is my first blog ever. I don't even know how to write a blog. I just wanna say "what I see in what I do". By starting my first blog technical never think I am a technical geek. I was on the road of thinking which can be my first blog. I thought it should be something that made me to pull my hairs but very simple. The below one is one of such rare piece and the most interesting problem I faced recently(later i know many developers faced the same :) ) which made me to Smile at myself when I found the solution. Let me start with the problem. I found that this post will really help others to spend time with their loved ones very FAST.
Recently I was working very hard(I think so) for my project suddenly came across a problem JVM started scolding me Java IO Exception: Too many files open .
I never saw this error while testing at developer side. But when I increased the number of source files, JVM started throwing exception. My JVM now Became very rich with the file I opened. As all other developers I tried to get hold of my friends hand (GOOGLE) to find a solution for the problem. You know whenever we ask for any help we will be given with tons of advice and its our intelligence to choose the best. My friends also showed my thousand solutions. Even a basic man a understand the problem occurs because numerous file descriptor are hold by the process at that point of time.
The solutions I tried
Increase Number of File descriptors
After a bit goggling i tried to increase the number of file descriptor (OS Limit) that each process can have. But later i started thinking Is this a correct solution????. My mind pushed me back asking a question Are you a good developer how can you expect user to change something in his system to run your code. Now i have to find the real solution to the problem.
Open Streams
More goggling left me with only one problem to look in to none other than closing all the streams I opened. Even Java provide with internal garbage collector (soon i have to write my own :) ) it is of no much use. So I started debugging my code finding for all open streams and closing them. Not only the file stream we have to close all the socket connection as the socket in-turn open stream underneath. To close a socket you simply need to close the underlying stream. Again I tried running the same program now the exception is thrown after processing some more files which gave me confidence that there are still opened streams in code but my bad luck I can find none. So again my friend came for help. This time eureka I found something interesting, actually my code is invoking some process on runtime.
One point all developers has to be careful here is even when you are not doing anything with the streams of the process it is your responsibility to close them. For ex say
Proces p;
//... initialize the process.
try{
p.getInputstream().close();
p.getOutputstream().close();
p.getErrorstream().close();
}catch(Exception e)
{
System.out.println("Error while closing the process stream");
e.printStackTrace();
}
My bad luck even after trying this I not able to solve my problem. Then I started to debug my code line by line to find the culprit who is opening the stream and not closing the same. I used Linux lsof command to find the number of files opened by a process at any time. Yahoo!!!! I got the culprit who made me to SMILE AT MYSELF. The culprit is none other than one of the reader I used from the existing source code thanks to one who developed which I never expected to cause the problem. Everything started going fine once i closed the reader. So all be careful that even when you close all the stream you open, there can be some stream (Input/Output/Error) left open by any one of your Senior HELPER coder (Is that really a great helper code. Left with you to answer). Close all the stream go to your pantry order some cookies and a drink spent Night time in your own way ;).
This is my first blog ever. I don't even know how to write a blog. I just wanna say "what I see in what I do". By starting my first blog technical never think I am a technical geek. I was on the road of thinking which can be my first blog. I thought it should be something that made me to pull my hairs but very simple. The below one is one of such rare piece and the most interesting problem I faced recently(later i know many developers faced the same :) ) which made me to Smile at myself when I found the solution. Let me start with the problem. I found that this post will really help others to spend time with their loved ones very FAST.
Recently I was working very hard(I think so) for my project suddenly came across a problem JVM started scolding me Java IO Exception: Too many files open .
I never saw this error while testing at developer side. But when I increased the number of source files, JVM started throwing exception. My JVM now Became very rich with the file I opened. As all other developers I tried to get hold of my friends hand (GOOGLE) to find a solution for the problem. You know whenever we ask for any help we will be given with tons of advice and its our intelligence to choose the best. My friends also showed my thousand solutions. Even a basic man a understand the problem occurs because numerous file descriptor are hold by the process at that point of time.
The solutions I tried
After a bit goggling i tried to increase the number of file descriptor (OS Limit) that each process can have. But later i started thinking Is this a correct solution????. My mind pushed me back asking a question Are you a good developer how can you expect user to change something in his system to run your code. Now i have to find the real solution to the problem.
Open Streams
More goggling left me with only one problem to look in to none other than closing all the streams I opened. Even Java provide with internal garbage collector (soon i have to write my own :) ) it is of no much use. So I started debugging my code finding for all open streams and closing them. Not only the file stream we have to close all the socket connection as the socket in-turn open stream underneath. To close a socket you simply need to close the underlying stream. Again I tried running the same program now the exception is thrown after processing some more files which gave me confidence that there are still opened streams in code but my bad luck I can find none. So again my friend came for help. This time eureka I found something interesting, actually my code is invoking some process on runtime.
One point all developers has to be careful here is even when you are not doing anything with the streams of the process it is your responsibility to close them. For ex say
Proces p;
//... initialize the process.
try{
p.getInputstream().close();
p.getOutputstream().close();
p.getErrorstream().close();
}catch(Exception e)
{
System.out.println("Error while closing the process stream");
e.printStackTrace();
}
My bad luck even after trying this I not able to solve my problem. Then I started to debug my code line by line to find the culprit who is opening the stream and not closing the same. I used Linux lsof command to find the number of files opened by a process at any time. Yahoo!!!! I got the culprit who made me to SMILE AT MYSELF. The culprit is none other than one of the reader I used from the existing source code thanks to one who developed which I never expected to cause the problem. Everything started going fine once i closed the reader. So all be careful that even when you close all the stream you open, there can be some stream (Input/Output/Error) left open by any one of your Senior HELPER coder (Is that really a great helper code. Left with you to answer). Close all the stream go to your pantry order some cookies and a drink spent Night time in your own way ;).
