Joseph "yields, rather than rejects. He releases (apolūsai) Mary, rather than casts her out (apolūsai)."

Annunciation to Joseph

"Joseph [was] 'just' [not in the spirit of Caiaphas, but] in the prophetic spirit of [John] the Baptist, and thus awed by the mystery he faced and respectful of its remove from the ordinary. . . . [H]aving been told by Mary of the Annunciation as soon as it took place, and thus well before any outward sign of Mary's pregnancy became evident, [and] being at the same time a man imbued with a profound sense of surrender to God’s plans and . . . an equally profound sense of that fear of God that precludes divulging the depth of a mystery, [Joseph] decide[s] to remove himself from the scene as unworthy to be publicly associated with the unfathomable” (92-93).

"Joseph accepts Mary's report as to how she had conceived, and thus, far from being scandalized by assuming adultery, he is profoundly respectful of a divine intervention that seems to preempt his role as husband.  He yields, rather than rejects.  He releases (apolūsai) Mary, rather than casts her out (apolūsai)" (70).

"Joseph, having accepted at face value and with deep trust the young girl’s revelation that she had become pregnant through divine intervention, is so awed by her and the mystery she (literally) embodies, that he feels he is confronting God himself, in front of whom he should retreat and hide in his nothingness" (69).

But "The vision of the angel in a dream points him in a different direction:  he is indeed a part of the mystery, and his role is to convey Mary, without delay, into his household" (63).

     Giorgio Buccellati, "The prophetic dimension of Joseph," Communio:  international Catholic review 33, no. 1 (Spring 2006):  43-99.  Cf. John Saward, Cradle of redeeming love:  the theology of the Christmas mystery (San Francisco:  Ignatius Press, 1992), 205-206, where St. Thomas Aquinas, St. Bridget, and Dionysius the Carthusian are all quoted respectively, as follows:

Joseph had no suspicion of adultery, for he was well aware of Mary's chastity.  He had read in Scripture that a virgin would conceive. . . . [H]e also knew that Mary was descended from David.  It was easier, therefore, for him to believe that this had been fulfilled in her than that she had committed fornication.  And so, regarding himself as unworthy to live under the same roof with someone of such sanctity, he wanted to put her away privately, as Peter said, 'Leave me, Lord, for I am a sinful man' (Lk 5:8) [(Lectura super Matthaeum cap. 1, lect. 4)]. 

Secundum autem Hieronymum et Origenem non habuit suspicionem adulterii. Noverat enim Ioseph pudicitiam Mariae; legerat in Scriptura quod virgo concipiet, Is. VII, 14 et cap. XI, 1: egredietur virga de radice Iesse, et flos de radice eius ascendet etc.; noverat etiam Mariam de David generatione descendisse. Unde facilius credebat hoc in ea impletum esse, quam ipsam fornicatam fuisse. Et ideo indignum reputans se tantae cohabitare sanctitati, voluit occulte dimittere eam, sicut Petrus dixit: exi a me, domine, quia homo peccator sum, Luc. V, 8. 

[A]fter I gave my consent to God's messenger, Joseph, seeing my womb enlarged by the power of the Holy Spirit, was exceedingly afraid.  It is not that he suspected me of anything untoward, but simply that he remembered the words of the prophets when they foretold the birth of the Son of God from a virgin, and reckoned himself unworthy of serving such a mother, until the angel in a dream commanded him not to be afraid but to serve me with charity [(Revelations lib. 7, cap. 25, new ed. vol. 2 [Rome, 1628], p. 239 f.)]. 

There is no doubt that the interior grace, sanctity, and chastity of Mary shone forth wonderfully and powerfully, not only in her face, but in the bearing and deportment of her body, so much so that anyone diligently considering her manner of life could not suspect her of fornication or any other sin [(Enarratio in evangelium secundum Matthaeum cap. 1, a. 3; DCOO 11:17AB)].

Cf. Cavadini.