I am unwilling to mix my fortune with him that is going down the wind.

Pepys’s metaphor of a man “going down the wind” evokes a vessel running before the wind—moving fast, but also potentially out of control and headed toward danger. The speaker’s refusal to “mix my fortune” with such a person expresses a cautious, self-protective ethic: do not entangle your prospects, money, or reputation with someone whose course is reckless or whose decline seems inevitable. In Pepys’s world of patronage, office-holding, and shifting political tides, the line captures a pragmatic awareness that alliances can be liabilities as well as assets, and that survival often depends on judging when to detach from failing ventures or compromised associates.