He bought it for a song in 1984 compared with what his fellow financiers were spending on tonier Park and Fifth Avenues.