Lancaster Castle, parts of which date back to the 12th century, is thought to be the only prison and court still operating in a medieval castle.
But the building in the city centre is also a popular tourist destination and theatre venue, which is said to have become outdated and too expensive to maintain.
Talks are currently taking place between the Ministry of Justice, Lancaster Council and the Duchy of Lancaster, which owns the castle, about ending its use as a prison.
A spokesman for the Ministry of Justice, said: “We have entered discussions with the Duchy of Lancaster and the local council about the future of HMP Lancaster Castle because it is outdated and expensive to run.
“As the previous Chief Inspector of Prisons made clear, it is difficult to run a modern training prison in a medieval castle.
“If an agreement is reached to return the prison to the Duchy of Lancaster, which owns the site, we will relocate directly employed NOMS staff to nearby establishments and make arrangements to place prisoners in more suitable accommodation.
“We need prisons to be places where staff are able to effectively rehabilitate prisoners to stop them re-offending and where capacity requirements are met more efficiently to improve value for money for the taxpayer.”
But Steve Baines, a national executive committee member of the Prison Officers’ Association, said it would be wrong to close the prison.
He said: “We are a bit disappointed they are looking to close what is a high performing prison.
“It is the highest performing Category C prison in England and Wales and the fifth highest performer in total.
“It has had a positive chief inspector’s report and it is achieving all its targets.
“What they are saying is because it is in a castle, it is perhaps not fit for purpose, but if it is not fit for purpose why is it performing so well?”
Mark Cullinan, chief executive of Lancaster Council, said: “In 2006 the Lancaster and Morecambe Vision Board, of which Lancaster Council is a member, launched a consultation on the vision for the future economic prosperity of the district.
“It recognises that Lancaster Castle has the potential to be transformed into a major tourist attraction which could attract significant complimentary private sector investment and that this would be a highly valued development for Lancaster.
“Even though we live in difficult times, Lancaster Council strongly supports this vision for Lancaster Castle and is enthusiastic about contributing to talks that can take this forward.”
Among the most well-known historical figures to have been held at Lancaster Castle were the Pendle Witches almost 400 years ago, who have since been the subject of various films.
In 1612 10 women were sentenced to death and hanged on the moor above the town. Others were given prison sentences. Father Edmund Arrowsmith was held there before his execution after he held a Mass in 1628 and George Fox, founder of the Quaker movement, was also held in the castle in 1664.
Source: Lancaster Evening Post