The reason why it bothers is that these geeks want to provs they are geekier than you by MEMORIZING something!
Geekery is about PASSION, memorizing takes dedication, but doesn't prove you're a good fan! I know some trivia about Hitler, like that he was a mediocre artist, does this make me fan of him? I certainly hope not!
It's not only the fact that these people think trivia makes them geekier than you, it's their boast of knowledge what proves their ignorance!
So I have to lose my "geek cred" because I don't know what Spock had for breakfast in a certain episode of Star Trek? Do you guys have ANY idea how much Sci-fi is out there? Claiming to know all about Sci-fi is claiming you're ignorant of how much there is out there! And it's even possible to be a Sci-fi fan and (GASP!) never have seen Stark Trek or Star Wars!
This applies to ALL areas of geekery, videogames, comics, fantasy...
Also, think about who would be more deserving of praise or "geek points", the geek who spent 20 hours watching a TV show memorizing it, or the geek who made a fan film with his friends? The geek who spent 2000$ on some actor's underpants or the one who created a piece of art about the series or movie the actor was in? The geek whose at every con flaunting his knowledge or the geek who brings sandwiches and juice?
Socrates said it better than I could ever say it: "All I know is that I know nothing"
I'm not saying knowing trivia is bad, but once you assume you know everything and think you're better than the rest... you prove you're not... There will always be something new to learn about what you love, and that should be enough for you!
So the next time someone tells you should have your "geek card" revoked, tell them that real geeks don't need them!