Quick Quote: Why Expert Systems Fail

I finally had a chance to read “Why Expert Systems Fail” by Michael Z. Bell. Its an old one and oft referenced, but hadn’t read it yet.

Many of the concerns it raises aren’t limited to just classic ‘expert systems’ but really to any decision-support tool.

• user acceptance (including too much/too little trust, legal liability, and so on)
• testability (including lack of consensus on correctness, scope, extraordinary events, and such)
• knowledge representation (possibly the least currently relevant)

Money quote:

Expert system development is as fraught with problems as any software development: life is not any easier because of an ‘artificial intelligence’ or ‘fifth generation’ name tag. If anything, it is more difficult. The major problems which are not as common in ‘conventional’ software developments are those concerned with the eliciting of expert knowledge, avoiding ‘common sense’ errors and verifying that the system will perform as intended. The most critical of these is the ‘closed-world’ problem the problem of the system knowing what it does not know. A little knowledge is a dangerous thing and may be worse than no knowledge at all. Think carefully, therefore, before removing the human from the system: he has far more common  sense and is therefore far better able to deal with the closed world assumption than any expert system is ever likely to be.

Bell, M. Z. (1985). Why Expert Systems Fail. The Journal of the Operational Research Society, 36(7), 613–619. https://doi.org/10.2307/2582480