Pigeon Key is a small island and at just 5-acres is home to 8 buildings on the National Register of Historic Places, some of which remain from its earliest incarnation as a work camp for the Florida East Coast Railway. The former Assistant Bridge Tender's House has been converted into a small museum featuring artifacts and images from Pigeon Key's colorful past.

The island was originally known as "Cayo Paloma" (literally translated as "Pigeon Key") on many old Spanish charts - said to have been named for large flocks of white-crowned pigeons (Columba leucocephala Linnaeus) which once roosted there. During the building of Henry Flagler's Overseas Railroad Key West Extension between 1908 and 1912, there were at times as many as 400 workers housed on the island. While these workers built many bridges along the route through the lower keys, the Seven Mile Bridge, spanning the gap between Knight's Key and Little Duck Key remains the largest and most impressive component of what was once referred to as "the 8th Wonder of the World". A number of buildings from the Flagler era remain on the island and are now part of the Pigeon Key Historic District.