Why do the ghosts of Henry VIII's wives haunt houses that didn't exist in their lifetimes?

Castle Lodge Ludlow is reputed to be haunted by the ghost of Catherine of Aragon who lived in Ludlow briefly following her marriage to Arthur Prince of Wales until shortly after his death in 1502.

The Queen's House, Tower of London is reputed to be haunted by the ghost of Anne Boleyn who stayed in the Tower's Queen's House prior to her coronation in 1533 and prior to her execution in 1536.

Yet neither building existed during the lifetime of Catherine or Anne as it does in the present day.

Castle Lodge was actually built sometime after 1571 when it was leased to Thomas Sackford, it was described in 1587 as "a fayre house of Maister Sackford which he did bylde" it does however have a stone doorway and medieval moudings that may be from an earlier house on the site.

The Tower of London's Queen's House isn't even on the same site as the Queen's House that Anne Boleyn stayed in, that was demolished in the eighteenth century. The current Queen's House was built around 1540 about four years after Anne's execution.

So why do the ghosts of Henry VIII's wives haunt houses that didn't exist in their lifetimes?

Are these hauntings examples of people having jumped to the conclusion that the apparitions must be those of well known historical characters who were believed to have once visited the buildings? What do you think?

Castle Lodge Image:User Rept0n1x at Wikimedia Commons
Queens House Tower of London Image: Cynborg  licensed under the Creative Commons Attribution-Share Alike 3.0unported license