Uh, you’re forgetting that Metatron also said it was the last piece as part of a spell. Who knows what he had to do with it? It might very well be gone. Plus, what happened to Metatron? One has to assume he didn’t fall with the other angels, so maybe he’s holed up in Heaven behind the locked doors? Inaccessible.
So this very well might be a whole new chapter for Cas. Being human, never to be an angel again. The implications of the show becoming what we saw in S4’s “The End” is really interesting and exciting (if they go that way). Honestly, I’m excited to see how this new Castiel will be without his angelic powers.