If you think of backlinks like a game of Jenga - each link representing a block. The more blocks/backlinks you keep, the more stable your structure/rankings. If you add more blocks/backlinks, your structure/rankings will rise. However, once you begin to remove blocks/backlinks, your structure/rankings become less stable, and you run the risk of collapsing/dropping your structure/rankings.

So, my advice to you is to keep adding more blocks.