I'm not sure if I have a problem with HARD games, but when I have to stop and ask myself if I'm still having fun, then I usually am not and that's where I draw the line.
For instance, the Gears of War and Halo games on the hardest difficulty are all pretty damn tough, but I think their campaigns still remain pretty fun and entertaining on those settings. Plus there are other achievements to be earned and reasons to complete them on harder settings. The same goes for most games now in that regard.
There are some games though that kind of slap gamers in the face with difficulty, whether it be development related flaws or actual just all out hardcore difficulty level. It seems like Capcom makes their games with no real regard for their gamers. A lot of their unlockables and aspects of mastering the game are just ridiculous life ending stuff. I'm speaking of the early Resident Evil games and Dead Rising in particular.
I also got really frustrated with MGS3 on Xbox 360 recently, trying to shoot all the green frogs, which was easy until the last 8 or so when you're on a motorcycle going 120 mph.
Games don't have to be hard and miserable, they can be hard and fun as well. Take Plants vs. Zombies for instance. That game never stops being fun even if you're getting your ass kicked on one of the mini games. It's just fun and entertaining like games should be.