Why are the rights of convicted felons still abridged even after their sentence is served?

There’s been a lot of talk lately about the rights of felons because of changes to the law in Virginia.
It made me wonder why we started limiting the rights of ex-felons in the first place?

Lawmakers debate felon voting rights restoration