I am not against golf, since I cannot but suspect it keeps armies of the unworthy from discovering trout...

The speaker affects tolerance for golf not out of admiration, but because it conveniently distracts “armies” of people he deems “unworthy” from encroaching on a more cherished, quieter pursuit: trout fishing. The humor depends on mock-elitism and the idea of scarce, fragile pleasures—wild trout streams—being threatened by crowds. Beneath the joke is a familiar conservationist/angling sentiment: that solitude and unspoiled waters are part of the sport’s value, and that mass participation can diminish it. The line also plays on the cultural contrast between golf (social, manicured, status-coded) and trout fishing (rustic, skill-and-patience-coded), using that contrast to satirize taste and class.