"Con" here means "jerk, fool, idiot".
It means "cunt" in an clearly sexual context, and as in English, the two meanings overlap when the word is used by someone who thinks "female" automatically means "stupid".
But in the song Brassens is saying that people of all age groups can be foolish, and the young and the old usually take each other for fools.
There was a popular french film called "Diner de Cons" which illustrates this pretty well. A group of rich guys invite wacky eccentrics to dinner for the fun of watching them make fools of themselves.
Just took a crack at making it more colloquial English rather than a word for word translation. Someone else could try ot make it a song!