If you are just filing down the sides so the buttons fit then a rounded file would probably be the best and you would want something thats nor very course. Otherwise you may end up chipping the lexan. As for using sand paper to smooth it out. I done thing you will really need that. Once the button is in you arent going to see it and if your file is fine enough it wont be that rough.
I did something similar when I made this stick
It has a black plexi top with a while plexi insert. I drilled out most of the rectangle and then filed it to size. I didnt have to clean up the hole at all when I was done.
In that picture you have of the files, I used one like the triangular one, but then I had corners in mine so that worked out better.