So a guilty Matthew will leave Mary at the slaughterhouse at the end of S2. She marries Sir Richard, who will only abuse her further both physically and psychologically, destroying her spirit. So Mary deteriorates, since she has no one looking out for her at Haxby. Carson made it clear he would rather stay with her womanizing father and Anna starts a new life of her own with the troubled Mr. Bates.
So why would Matthew reenter her life again? To rescue her from her evil husband? If he isn't man enough to prevent her from getting married to that perv, then what's there to make him save her now, when it's too late?
Or would Matthew investigate the Pamuk story (and Vera Bates' demise and Sir Richard's role in that)? What would his unearthing that do this time? Put Sir Richard in prison for Vera's murder? And even so, wouldn't Mary stick by her husband, if only to protect her reputation? Wouldn't Matthew's interfering make things worse for Mary and even be detrimental to her relationship with him? I mean, unless the two of them actually elope and run away out of England, they won't really be safe, would they? And of course, Matthew would be the last person on earth to do something so unconventional. Oh no, he's no Branson.
So if we take the huge time jumps into account, Mary will be in her thirties by this time and will probably be saddled with children, which would make it impossible for her to split from Sir Richard. But Fellowes may indeed kill him off and set her free. But not really, because Matthew would lap after her like a puppy, only to bite her in the cheek.
And Matthew, would he be in life-long mourning for Lavinia? Or, as is typical for him when he can't deal with his feelings, does he get himself engaged to yet another insipid woman within less than two months' acquaintance?
Regardless, so he rescues Mary first (because a hero has to, of course), befriends her children, and raises her hopes again. But then he is paralyzed by some sort of affliction that resembles "honor" and he leaves her again.
Cue S4.
And S5.
And S6.
.
.
.
And Fellowes decides to be generous with us fangirls (who have aged and moved beyond tumblr).
So Mary's son and Matthew's daughter marry. And lay flowers on the graves of their parents. Who are buried near each other.
The end.
Or maybe not. Patrick returned after all.