Heat oven to 350 F.
Spray pan with vegetable spray.
For the plums, place 6 Tablespoon butter in an 8-inch-square pan, or 9-inch-round, 2-inch deep cake pan, or an 8-inch springform pan. If using springform pan be sure to wrap the outside with foil. Place the pan in the oven to melt the butter, 4-5 minutes. Remove pan from oven; stir 1 cup brown sugar and brandy into the melted butter in pan.
Overlap plum slices in rows if using a square pan or in a circle around the edge if using a round pan. Repeat in smaller circle in center to cover the bottom of pan.
For the cake, whisk together flour, baking powder, cinnamon, and salt.
Cream softened butter in a large bowl with electric mixer until butter is light and fluffy, 5 minutes.
Add 1 cup granulated sugar; cream until sugar dissolves, about 5 minutes more. Add eggs and vanilla; beat until combined.
Add dry ingredients to creamed mixture alternately with milk, using a whisk to avoid overmixing. Whisk ingredients only until well blended, cleaning sides of the bowl with a spatula. Add the toasted almonds until combined. Spread batter evenly over plums in pan.
Bake the cake until a toothpick inserted in the center comes out clean, about 45 minutes. Let cake cool in pan on a rack, 30 minutes.
Run a knife around the edge of the pan; invert the cake onto a serving plate. Let cake stand 1-2 minutes, then lift off the pan. Glaze with the plum sauce.
While the cake is baking make the glaze by placing the plums and sugar in a saucepan. Cook over medium heat, stirring with a wooden spoon, until the mixture starts to boil. Add the lemon juice and continue to cook, stirring, for about 5 to 8 minutes, or until the fruit is glazed and a bit softened but not dissolved.
Pour the mixture into a bowl and add the vanilla and cloves. Cover and let the sauce come to room temperature.
Gently remove the plums from the glaze and place in a container to eat later on yogurt, ice cream, or just plain.
Glaze the entire surface of the cake.
Serve the cake warm with whipped cream or crème fraîche.