Sometime people (often it happens with good students) asks to me how to find vulnerabilities. This is a great question but unfortunately the answer might take forever. The short answer is to look for bugs. Indeed, a vulnerability is a specific bug. A vulnerability is a particular bug which affects the security of the entire system. So usually I say, find a bug, then come to me and I will show you how it's possible to transform some kind of bugs in vulnerabilities.
Well, the second question comes automatic, how do I find a bug ? Even this answer could fill up a whole book but usually I say: you might find a bug by doing fuzzing. Today I wanna introduce to you BAD: the Bruteforce Exploit Detector.
BED is a program which is designed to check daemons for potential buffer overflows, format string bugs et. al. BED simply sends the commands to the server and checks whether it is still alive afterwards.Of course this will not detect all bugs of the specified daemon but it will (at least it should) help you to check your software for common vulnerabilities.
BED is particularly good for remote fuzzing while if you need "something more local", testing local parsing file bug I do always suggest gruba. I know ... those are pretty old and now there are "universal" fuzzers including PE, MACHO, ELF, HTTP, FTP, UDP, SNMP and so forth and so on... but both BED and gruba are pretty easy to analyze and, eventually, to expand.