Update:
I took off the vapor canister to weigh it. As I had it in my hands I noticed that 2 of the outlets on it were packed with charcoal(fuel tank & purge). I have no idea how one of these work but I assumed it wasnt suppose to be that way. So I took a screw driver and cleared each hole. Then I tipped it over and dumped the loose charcoal into a bottle. Then after cleaning everything I reinstalled it and went to the gas station. I filled up the tank without it popping off once. So it appears that the canister was the cause of the problem. Does this mean I should get a new one or is that just something that happens?