At the moment it is not possible to sell or delete buildings in the game, but you can move them.  However, it is possible to sell any of the decorations at a reduced price.  Simply click on the item that you wish to sell, and select "sell":

Please note that if you sell an item, you won't receive the full purchase price back. Only a fraction will be restored to your account, as stated by the confirmation pop-up. 

For instance, the Weather Vane can be bought for 10,000 elixir, and you can can sell it back for 1,000 elixir.