You can get the drawings you need from an “architectural designer”. So not necessarily from an architect. But what do those drawings show? They show a design. You are more likely to get the highest quality design from an architect. Because their capability and training is more extensive than an “architectural designer”.
Some architects are not as good at providing drawings for construction as some “architectural designers”, especially architectural technologists. And some architects create designs which are unnecessarily hard to build. So your builder may have struggled to work with such an architect. However most architects do work successfully with good builders.
Ask your builder why he thinks you don’t need an architect. Ask your architect to reassure him. But some builders simply won’t engage. Because they want to cut corners for profit. Which you may not notice. But an architect will. Part of the architect’s duties is quality control. So beware of builders reluctant to communicate with an architect.
At best those builders want to take full control to their own advantage. Not yours. At worse they may be “cowboy” builders leading to an unlawful building. Of poor quality. And one costing you far more than it should.