By analogy of this, and cardinal points, cardinal winds, cardinal signs (four zodiacal signs marking the equinoxes and the solstices), the adjective in Middle English acquired an association with the number four.
early 12c., “one of the ecclesiastical princes who constitute the sacred college” (short for cardinalis ecclesiae Romanae or episcopus cardinalis), from Latin cardinalis “principal, chief, essential” (see cardinal (adj.)).
Ecclesiastical use began for the presbyters of the chief (cardinal) churches of Rome. The North American songbird (Cardinalis virginianus) is attested from 1670s, so named for its resemblance to the cardinals in their red robes.