I think dealers need to be involved somehow because I don't see the majority of people willing to order a car sight-unseen and without taking a test drive. Purchasers of Teslas (and EVs in general, now) are usually a different breed. It would be nice to have some type of hybrid system where you order your specs based on the manufacturer's price and the dealer gets a cut for handling the test drive, paperwork, and delivery. I don't know who would pay for the cars sitting on the lots available for test drives. Also need to be able to get a car quickly to someone whose car died and was totaled. Dealers make most of their money on service and used car sales, anyway, so I think they'd do OK.
I'd also love to go to one location where I can look at, for example, a Honda CR-V, a Toyota RAV4, and a Nissan Rouge and have someone who was knowledgeable with all three. Right now, I go to a Toyota dealer and they tell me that the turbo on the CR-V won't last, the Honda dealer tells me the RAV4 is loud, etc. A salesperson should help you find the car that suits you, not try to shoehorn you into whatever they are selling!