Guidance needed on recaulking a bathtub (old silicone caulking)

We are in the process of scraping out the old caulking in our tub. What I need to know is whether or not we have to get every little bit of silicone caulking out before we redo the caulking (with silicone)? This has been quite the process as the last guy who did the recaulking left half of the old caulking behind.

There are only tiny bits of caulking left here and there and I just don’t want to go crazy trying to get the last of it out (it is highly resistant to being taken out even after having the silicone caulking remover coated on it).