You could cover it with a Outlet Cover with a flip cover on it should you need to get at it. May need to do a little minor modification to the cover to allow the wires to feed out between the back of it and the wall though.
Well I just implemented Jack L's idea with some foam I had for underneath soft pool coping. We'll see if it works! Tried to wedge the foam between the switch and the hole in the wood, but it was slippery and hard to fit both arms and head to work in the closet. My first "modification".