Throughout its history, Cape May has played host to a range of influential American figures, including several Heads of State. Distinguished visitors include Henry Clay, Franklin Pierce, James Buchanan, Ulysses Grant, Chester A. Arthur and Benjamin Harrison, the latter of whom named Cape May's Congress Hall Hotel his "Summer White House" and conducted the affairs of state from inside the walls of the hotel. Many musicians and cultural celebrities also paid visit to the town such as John Phillip Sousa, who frequently conducted his band on the lawn of Congress Hall.

The second half of the 1800s saw a period of turmoil for the resort town. The onset of the 1860s brought with it the national strife of the Civil War, which polarized the city's Southern plantation owner and Northern industrialist clientele. (Ultimately, Cape May aligned itself with the Union with rapturous enthusiasm at a December 1861 city hall meeting.)

In 1878, a massive fire broke out of the Ocean House on Perry Street. Cape May never stood a chance: its entire fire-fighting apparatus consisted of a single hand engine, 1,500 feet of rubber hose, three chemical units and a hook-and-ladder truck. A train was dispatched from Camden, 90 miles away, with a steam engine and a supply of hose, but by the time it arrived, four hotels, including Congress Hall, were ablaze. Fanned by a 36mph wind from the west, the fire spread further east, and by the day’s end early 40 acres of prime beachfront property had been completely destroyed.

From the ashes of the great fire, Cape May was able to rebuild. Prominent architects of the period flocked to the city with commissions to craft Late-Victorian masterpieces, each more grand than the last. One such architect was Stephen Decatur Button, who designed over 40 buildings in Cape May including the 1891 Italianate-style Atlantic Terrace Houses off of Jackson Street (four of which now make up the Virginia Cottages). Other architects credited with work in town include Frank Furness, Samuel Sloan, and Enos Williams.

Cape May continued to forge its place in history throughout the twentieth century. The island served as a key military station during World War II (relics of which include a lookout tower by Sunset Beach and a cement bunker on the beach by the Lighthouse.). While many neighboring shore towns opted to tear down many of their historic structures in favor of new development, Cape May recognized the value in its Victorian past. In 1970, the Historic District was added to the National Register of Historic Places, and in 1976 the town was listed as a National Historic Landmark.

Today, Cape May is in the upswing of a renaissance that uses the past to energize the present. The famous Gilded-age resorts and beach cottages have been restored to their former glory, and fresh and creative boutiques and restaurants can be found on every corner. Generations of vacationers have been charmed by this place over the centuries. Walking through the streets as a breeze blows in from the Atlantic, it's easy to understand why.