He had to hide it for a long time, so he just kept it in his mouth; the safest, most accessible place. When the time was right, he took it out.
It's not clear what he did with the rest of the pen, or when and where he did it. Maybe he ditched it before leaving Baltimore.

But it was a metal pen.
That's why I though eating it was implausible.
I haven't read the book so I don't how Thomas Harris explained it.
If he was intending to kill the guards, why not break off the part of the pen he would use to pick the lock, hold it in his hand and then leave the rest of the pen behind after he escapes?

It's the right answer though.
My password is password.