Cover potatoes with water by 1 1/2 inches in a 2-quart pot, and bring to a boil. Reduce heat, and simmer until potatoes can be pierced easily with the tip of a knife, about 20 minutes. Drain, and let cool completely. Cut potatoes into halves, or quarters if large.
Stir potatoes, creme fraiche, buttermilk, horseradish, salt, and pepper together in a bowl. Gently stir in herbs just before serving.