Sketching Geometry and Physics Informed Inference for Mobile Robot Manipulation in Cluttered Scenes