sem_remove() shouldn't be part of a normal cleanup/teardown and should be called very rarely due to bugs in the implementation.
It doesn't seem to hurt to leave semaphores lying around and is likely to be more performance-friendly to do so. If you are concerned about having too many semaphores floating around for an application (e.g. a file cache that uses ftok()), you can use some modulus arithmetic and simple addition to create a limited range of values for your semaphores off in the middle of nowhere. For example, ftok() % 101 + 0xBADBEEF. Be sure to replace 0xBADBEEF with your own random value. The example limits the range to 101 semaphores. 101 is a prime number - so if you want more or less, be sure to replace it with a prime number since primes theoretically help distribute values more evenly.