I used to think the same thing and then I got roped into some lessons and got hooked on it and I had it bad for a long time. Once I moved away from my usual golf crew I slacked off until now I almost never play,  I think comradery has a lot to do with it. You know your with good friends when you swing as hard as you can, grunt like a power lifter and the ball goes 2 feet in the wrong direction and nobody talks shit because they feel your pain.