Let's get one thing out of the way before I continue though. Sometimes atheists aren't upset at all, even when they're being told that they are upset. For instance, there's a story that was making the rounds online recently about how atheists are supposedly enraged over a song by Carrie Underwood. This has a lot of Christians wondering why atheists would get so mad just because somebody would sing about their religious beliefs. The thing to notice though is that no actual atheists have been quoted as being mad about the song at all. In fact, the only response I've been seeing is not a response to the song but a response to the original article. In other words, atheists aren't upset about the song, but they are baffled as to why they're being called upset.
There's also a lot of exaggeration out there about things that supposedly upset us. Some comedian had a whole bit about an atheist who got offended because somebody said "Bless you" after he sneezed. Is it possible that there are atheists out there who would take offense to that? I suppose, but I think that I'll go out on a limb here and say that most atheists don't care, and many of us still say it reflexively.
Still, sometimes atheists seem upset, and sometimes they genuinely are. While I'm not the first person to explain why this happens, let me give a few possible explanations:
1. We feel like we've been duped.
Imagine what it's like to be married for twenty years only to find out that the person whom you thought was faithful to you was actually leading a double life and was cheating on you the whole time. This is how many people who are just coming out of faith feel. We feel like this idea, which we trusted to be true, has turned out to be a colossal lie.
I realize that if you are a believer, your response is that it's not a lie, so we shouldn't feel that way. But that's not my point. Even if we're mistaken in our belief that it's a lie, the point is how we feel about it. Nobody likes to be tricked about something that's important, and what can be more important than the fate of your immortal soul (if there was such a thing)?
2. We feel like those closest to us don't care.
Oftentimes, when a person loses their faith, it's due to a long process that involved a lot of reading and soul-searching. When they reach the conclusion that there is no reason to believe in a God, they want to go out and share this information. Their motivation is the same as yours would be if you found out that a particular model of car would explode if somebody barely bumped it from behind. You'd want people who own this car to know the truth. At the very least, you'd want some sort of good explanation as to why this isn't the case. Now imagine that people responded by explaining away the evidence or completely ignoring all together; instead, they chose to keep driving that death-trap.
I realize that many theists feel this way about when they share their faith. They think that they have some important information to tell those who don't believe in their religion. I've even heard Christians say that they proselytize because they want to "save people". I have no doubt that their intentions are sincere, and if you're a Christian, then you should be able to understand this frustration perfectly. Now, the question as to who's actually right is another story, of course, but hopefully you can see what I mean.
3. People don't even attempt to understand what we're saying.
Let's stay with the defective car analogy. What if you told people about this car, and their responses went like this:
"Why are you so angry at this car manufacturer?"
"You just say this because you love bicycles."
"So, are you saying that mayonnaise isn't healthy?"
None of those things have anything to do with a car that will explode, do they? How frustrated would you be if people responded this way? You'd probably start shouting and flailing your arms, I'd imagine. Well, this is how we feel when we get responses like this:
"Why are you angry at God?"
We're not. We'd have to believe in him to be angry at him. Many of us think that if he was real then he would be pretty awful, and it makes us angry that anybody would want to worship such a thing - but this idea can coexist with not believing that he's real.
Also, we're sometimes angry at believers because they want to enforce laws based on what he supposedly wants, but they aren't even able to demonstrate his existence in the first place. (And oftentimes, what this god wants is to treat certain groups of people unfairly.)
Do you want to make an atheist happy? Then respond as follows, and it doesn't require you to compromise your own beliefs:
"So, you don't think that a God exists, and you don't see any good reason to believe in one?"
From there, don't assume that they haven't considered the reasons why you believe. Chances are good that they have, and they find those reasons entirely unconvincing. (And in some cases, you'll even learn that there once was a time that they found those reasons to be compelling.)
4. Believers often create a false narrative.
This idea is related to the first one. I've had people assume many things about me, like how I must have had some bad experience, or I don't want to be held accountable for my actions and would rather "live in sin", or the most frustrating: I don't believe because I'm biased.
These are all untrue, and the last (most frustrating) one ignores that I started out with a bias to believe in God. If a bias is that strong, then how was I able to change my mind? Certainly a bias can explain why people stick to a particular position, but it's false to assume that's the reason why, and I think that I can demonstrate why it's not a matter of a bias that has me feeling the way that I do.)
Perhaps this isn't intentional on the part of the believer, but these sorts of narratives - along with the ever-popular "You're just going through a phase" - enables the theist to completely dismiss what the atheist is saying without having to consider any of their actual arguments.
And that, along with all of the other reasons I mentioned, can be very frustrating. Frustrating enough to make us a bit angry.