So My AVR kept behaving strangely. Nothing worked and as soon as I touched the Crystal pins, the software started. So I went through the arduino schematics and found a 1 Meg resistor between the crystal pins. Once I attached that, then everything started working normally. Not sure what is the role of the resistor but things seem to work on two of my boards now.