It's been said a dozen times already, but the tinkering is the most fun for me. First you tinker with the rifle, then you tinker with the load.
That said, I do think most people seek a level of accuracy that goes well beyond their shooting ability. A very small percentage of hunters are competitive target shooters, and relatively very few shots are taken past 200 yards. All that tinkering is not necessarily utilitarian or practical for most folks, but that sure doesn't mean that it aint one of the better parts of gun collecting.
You know you've got a gun problem when you've spent more money on guns preparing for a hunt than you did on the expedition itself. Some guys will buy a $30,000 english double rifle and then bitch that they don't have the funds to get to Africa. My plan, take the double (mines just a Chapuis and no where near that expensive) shoot it up, and sell it to pay for the trophy fees.