Did you know that the 32nd President of the United States was born, lived in, and was buried right here in the Hudson Valley? It is believed that the central portion of the present-day house is formed by a large farmhouse which was constructed around the year 1800 in the Federal style. In 1845, the estate was purchased by Josiah Wheeler, a merchant from New York City who remodeled the house, giving it a then fashionable Italianate style with a three-story tower at the south end as well as front and rear piazzas spanning the entire length of the house. After this remodeling, the house comprised a total of 15 rooms.

In 1866, the estate which had been reduced to approximately one square mile, was bought by James Roosevelt, Franklin D. Roosevelt's father for $40,000. The property featured a stable and horse track, which was important to James because he took great interest in horse breeding. From the date of purchase until his death 34 years later in 1900, James made many improvements of the house and property. He enlarged the servants' wing of the building, adding two rooms, and had a spacious carriage house built in the vicinity.