Good news, then! Your problem is not with anarchist. Not in the no true Scotsman way, either. I'll quote a good one at the end, but essentially, anarchists do not believe there should be no authority, anarchists believe that authority should justify itself. Just and unjust authority exists, regardless of how you want to organize society. Anarchists even have groups, parties, organizations. The question isn't whether or not we should have authority or whether we should just go buck wild and live every man for himself. What anarchists desire* is to maintain the infrastructure of our society, with resources being distributed in a more equitable way, and the decisions being made by way of the people who will actually be impacted by it, and by whose labor it was created.

*For the most part, anti and post civ not withstanding

Does it follow that I reject all authority? Far from me such a thought. In the matter of boots, I refer to the authority of the bootmaker; concerning houses, canals, or railroads, I consult that of the architect or the engineer. For such or such special knowledge I apply to such or such a savant. But I allow neither the bootmaker nor the architect nor savant to impose his authority upon me. I listen to them freely and with all the respect merited by their intelligence, their character, their knowledge, reserving always my incontestable right of criticism and censure.

That quote is generally referred to as the authority of the boot maker and most anarchists would suggest not only not referring to the boot maker when you need a tooth pulled, but also having the right, and often moral duty, of consulting more than one expert when possible