Binary shapes, or silhouettes, are essential in human communication. They include, for example, all fonts and many logos. They can be extracted from images in raster form but require a vectorization for resolution independent editing. In this paper, we propose a mathematically founded silhouette vectorization algorithm, which converts a raster 2D shape to a Scalable Vector Graphics (SVG) format whose control points are geometrically stable under affine transformations. The proposed method can also be used as a reliable feature point detector for silhouettes. Compared to state-of-the-art graphics software, our algorithm shows a superior reduction in the number of control points for an equal or better accuracy.