Lafayette Square “was once the most prestigious place to own a house in the city, and the rich and famous lived there,” emails John DeFerrari, a trustee of the DC Preservation League and author of the blog Streets of Washington. Now, those homes that flank the White House are mostly used for offices, as well as the White House Historical Association’s museum store, according to Ghosts of D.C. author (and former White House staffer) Tom Cochran. Thus, the government-owned estates will probably never be sold back to regular folk.

According to a 1957 article in the Washington Post—headline: “Washington’s Haunted Square”—celebrity dwellers at the height of the neighborhood’s prestige included glamourkittens like Montgomery Blair, Abraham Lincoln’s postmaster general, and historian George Bancroft, who famously developed the “American Beauty” rose in his garden. But the biggest star to live on the block was probably Commodore Stephen Decatur. As for the value now, it’s hard to tell, but you can rent the elegant courtyard of the Decatur House for eight or so hours for a fee starting at $3,000.