I think it’s not a bad idea to ask for some terms before a negative vote could be cast (like asking for short explanation about the problem). However, not everyone is able to express himself in English or wants to put time in formulating the problem.
One option could be to add a small list with predefined common problems, the user then needs to tick one or more of those options closest resembling his problem and then the negative vote option becomes active (maybe add field other for option not in list). But probably the simple independent thumbs up or down is most easiest way for expressing good or bad experience and everyone can easily use that?
A system that would leave only the option to sort things out with the designer can also generate problems, things can easily get out of hand and lead to disputes and personal opinions clashing.
Also for designers, isn’t it a positive thing there is a certain alertness needed?
No need to be afraid of customers but just know they have something to say even if that is just a red or green dot.
We can endlessly debate about what could be counted as carelessness from designer part or user part or what is good or bad work but the masses are very good at sorting those things out, it’s not really up to a few people and hopefully not the designer himself to decide if negative vote is valid or not. It is perfectly possible one gets 10 positive votes on a product and one negative for no apparent reason at all, personally I don't see that as a problem as long as the positive out-way the negative votes with a fairly larger margin.
If one gets negative votes consistently then he probably need to change some things?
I'm in favor for the buyer rating in general, some problems are indeed hard to avoid and can be user related but that looks to me like no real reason to get rid of the rating option altogether. However, I do believe there needs to be some automated mail delivery system that can handle the feedback collection consistently, currently people don't seem to find their way so easily to the voting system (just very few cast a vote).
If just few people vote then the negative voters have the edge because they usually feel more motivated to vent frustrations and look for ways to be able to do that.
If things are just ok then most people do not feel an urge to really express that, certainly not if they need to activity do something like login, click tabs and locate something.