You'll soon be able to remove a Twitter follower without blocking them. Don't worry, the removed follower won't be notified that they've no longer on your list of followers.
Twitter’s upcoming new feature lets you remove a follower without blocking them