Whisk cornstarch and 1/4 cup guava juice in a bowl until smooth; set aside.
Bring the remaining 4 cups guava juice and the sugar to a boil in a medium saucepan, stirring until sugar has dissolved. Reduce heat. Stir in cornstarch mixture; simmer, stirring occasionally, until mixture is thick enough to coat the back of a wooden spoon, 4 to 5 minutes. Stir in lemon juice.
Pour mixture through a fine sieve into a bowl; discard solids. Divide among six 6-ounce ramekins or bowls. Refrigerate puddings, uncovered, 30 minutes. Cover loosely with plastic wrap, and refrigerate until set, at least 3 hours (up to 3 days).
To serve, top each pudding with 1 tablespoon each yogurt and almonds.