But some say, basing it on a tradition in the Gospel according to Peter, as it is called, or “The Book [Protoevangelium] of James,” that the brothers of Jesus were sons of Joseph by a former wife, whom he married before Mary. Now those who say so wish to preserve the honor of Mary in virginity to the end, so that her body, which was appointed to minister to the Word, which said, “The Holy Spirit shall come upon you, and the power of the Most High shall overshadow you” [Lk 1:35], might not know intercourse with a man after the Holy Spirit came into her and the power from on high overshadowed her. And I think it reasonable that Jesus was the first fruit among men of the purity that consists in chastity, and Mary among women; for it is not pious to ascribe to any other than her the first fruit of virginity. – Commentary on Matthew 10:17 (249 A.D).

Origen of Alexandria