Destroying galaxies (or not) with AGN feedback