Funding gaps and a lack of desire is putting a historic military site at risk while it’s “crumbling into the ground”.