Metaethics
Metaethics is one of the three branches of ethics usually recognized by philosophers, the others being normative ethics and applied ethics. It’s a field of study that tries to understand the metaphysical, epistemological and semantic characteristics as well as the foundations and scope of moral values. It worries about questions and problems such as "Are moral judgments objective or subjective, relative or absolute?", "Do moral facts exist?" or “How do we learn moral values?”. (As distinct from object-level moral questions like, "Ought I to steal from banks in order to give the money to the deserving poor?")
Metaethics on Less Wrong
Eliezer Yudkowsky wrote a Sequence about metaethics, the Metaethics sequence, which Yudkowsky worried failed to convey his central point; he approached the same problem again from a different angle in Highly Advanced Epistemology 101 for Beginners. From a standard philosophical standpoint, Yudkowsky's philosophy is closest to Frank Jackson's moral functionalism / analytic descriptivism; Yudkowsky could be loosely characterized as moral cognitivist - someone who believes moral sentences are either true or false - but not a moral realist - thus denning that moral sentences refer to facts about the world. Yudkowsky believes that moral cognition in any single human is at least potentially about a subject matter that is 'logical' in the sense that its semantics can be pinned down by axioms, and hence that moral cognition can bear truth-values; also that human beings both using similar words like "morality" can be talking about highly overlapping subject matter; but not that all possible minds would find the truths about this subject matter to be psychologically compelling.
Luke Muehlhauser has written a sequence, No-Nonsense Metaethics, where he claims that many of the questions of metaethics can be answered today using modern neuroscience and rationality. He explains how conventional metaethics or "Austere Metaethics" is capable of, after assuming a definition of 'right', choosing the right action given a situation - but useless without assuming some criteria for 'right'. He proposes instead "Empathic Metaethics" which utilizes your underlying cognitive algorithms to understand what you think 'right' means, helps clarify any emotional and cognitive contradictions in it, and then tells you what the right thing to do is, according to your definition of right. This approach is highly relevant for the Friendly AI problem as a way of defining human-like goals and motivations when designing AIs.
Further Reading & References
- Garner, Richard T.; Bernard Rosen (1967). Moral Philosophy: A Systematic Introduction to Normative Ethics and Meta-ethics. New York: Macmillan. pp. 215
- Metaethics in the Stanford Encyclopedia of Philosophy