Yes, I had to drill out the center of the square shaft from the inside to reduce surface area of shaft that is binding and allow penatrating oil into the inside of the stuck part. Go full size on the image and it becomes pretty clear. Drill bit was essentially same size as shaft but round so it only left the "corners".
Was able to gently tap out with a punch after that. Well sort of gently.