Rights come with responsibilities. It is therefore the responsibility of the citizens of every country to make wise use of their rights in a way that does affect the country as a whole. This is why this freedom is sometimes limited.
