Preheat oven to 350°F. Grease and line three 8-inch pans.
Whisk flour, baking powder, and salt. Set aside.
Beat butter and sugar until fluffy, about 3 minutes. Add eggs one at a time, then vanilla.
Steep Earl Grey contents and lavender in warm milk for 5 minutes, strain, then add to batter alternately with dry mix.
Divide batter and bake 20–25 minutes. Cool in pans 10 minutes, then transfer to racks.
For Swiss meringue: whisk egg whites and sugar over simmering water until sugar dissolves. Whip to stiff peaks and cool.
Beat in butter, salt, and extracts until smooth. If it splits, chill briefly and rewhip.
Stack layers with a thin crumb coat, chill 20 minutes, then finish frosting. Garnish with a few lavender buds if you like.