It's not really so much as a cover as the whole housing that comes out.
You can try to just pull it out gently with your fingertips, or sort of angle it out. I suppose that if the plastic has "baked" more than others through time/exposure to higher temps, it may be more brittle and break.
Yeah, DMCH even sells a kit for those, the door ones. The license plate ones are L=38mm festoons.
There was a huge improvement on my door lights in going from very old (5 years) LEDs, to modern ones. Very visible now during even sunlight.
Check out details for converting to LEDs at the doors and other locations, at my blog:
"100% LEDs..."
..and...
"Let LEDs light your wings"