The charming seaside town of Duxbury, Massachusetts became prominent before the arrival of the Pilgrims in Plymouth in 1620, serving as a favored gathering spot for the Wompanoag people for fishing and hunting. Named as Island Creek by the early Pilgrims, this area was cultivated during spring, summer, and fall, while winters were spent in Plymouth.

The land presently known as Miramar was granted to Myles Standish's military assistant, Lieutenant William Holmes in 1627. By 1705, it was acquired by Thomas Loring, a descendant of the original settlers of Hull. The next 300 years saw the Loring family and their kin retaining ownership of the property, which became renowned as the Bay Farm. The Lorings also gained fame as distinguished shipbuilders in Duxbury and other ports stretching to Boston, making significant contributions to the Union during the Civil War by constructing iron-clad ships.

In the early 20th century, the Lorings formed the Bay Farm Company in 1904 to streamline the ownership of the varied parcels of land, which had become rather complicated over the years. With the goal of establishing clear title and boundary lines, they submitted the lands to the Massachusetts Land Court. The Company also operated a thriving dairy farm across the street from what is now the Bay Farm Conservation area.

In 1945, Mrs. Elizabeth Belknap sold the property to the Societas Verbi Divini (SVD) or Society of the Divine Word, the world's largest Catholic missionary order. This scenic property, spanning over 30 acres, boasts improvements including a large manor house built in 1900 and ancillary buildings, perched on a hill with stunning views of the Bay Farm and Kingston Bay. It came to be known as Miramar, Latin for 'see the sea'.

The Divine Word transformed Miramar into a retreat center for priests and brothers returning from missions. The retreat center expanded in size and mission over the years. Funds from selling a seminary property across the street in the 1980s led to extensive renovations and additions to Miramar. Until 2022, the retreat center served a wide variety of organizations, welcoming over 6,500 annual visitors before the Covid pandemic.