Well, I was in the sewers section after I got up the ladder.
I'm assuming this part of the dungeon is part of the castle since It is masoned architecture again. I actually fell down twice in that area. The first time through the floor, which might not have been a bug since the floor looked unstable and the fall just brought me back to the lower cave surroundings.
Ahh, you're in the goblins area where the leader of the goblins is. It's not the castle
If you happen to remember where in that area you fell through (the area near the ladder is fine), it'd be helpful to know . I've probably tested that level more than any other, and haven't found the spot, so idk if I'd easily find it.