May be his father:
He is the grandson of the governor of New Jersey. Also, he is a direct descendant of the first graduate of West Point, Joseph Gardner Swift. Further, he was an aide to President Roosevelt. At the time, he was required to be a bachelor to have that position. Information supplied by his grandson, William Wrightson.