Have you read any John Stuart Mill at all? In my humble opinion the man is the father of modern liberalism (small 'l' liberalism, aka the one we all subscribe to in western society). If you have not, I think that you should. He has wonderful thoughts and conclusion on the codified use of force by the state and the soft use of public shaming in order to control behaviour and responses to issues. For example, to Mill it would be against a liberal ideal to demand that the state lock up anyone who offends you with words, but he would welcome people telling that person that they are no longer invited to such and such party next weekend because of his positions. Mill believes that civil society could regulate a lot of their own behaviour through daily interactions that are rooted in common sense and an understanding of needing to just get along together (much like what you wrote about to this guy about organizing a party for example). He thus draws a line that the state should only intervene and limit the liberty of another person when said person is doing something that restricts the liberty of another. That is a pretty high bar when we think about it, that would mean that Nazis would be allowed to parade in the state and the police would have no right to stop it (since marching causes no physical harm to people) and each and every one of us would be within our right to record the names of the people in the march and fire them, cast them out of social groups, etc.