Which punctuation, if any, is correct:
A. Did you ever ask Barb, "Hey, what's the matter with you?"
B. Did you ever ask Barb, hey, what's the matter with you?
C. Did you ever ask Barb, Hey, what's the matter with you?
D. Did you ever ask Barb, "Hey, what's the matter with you"?
Option A is the best way to go here.
The quoted sentence is Hey, what's the matter with you?. To insert this into another sentence, you can:
Did you ever ask Barb what the matter with her is?
As I stated at the beginning, the quoted sentence is set in stone already. Because of this, you can't drop the quote marks like in option B and C. This would imply an indirect quote, but the material doesn't fit the sentence anymore. Option D essentially changes the quote from a question to a statement, so it isn't ideal.
A is correct. The others are incorrect. A comma follows the attributive, "ask," and the words quoted must begin with a quote mark and then a capital. The quoted material ends with the appropriate punctuation mark and then the quote mark.
This question raises at least four distinct issues with regard to punctuation or capitalization: (1) whether a comma is appropriate before the embedded question; (2) whether the embedded question should begin with a capital letter; (3) whether the embedded question should be enclosed in quotation marks (either single or double); (4) whether the end punctuation should appear inside or outside the quotation marks, if quotation marks are used. As it happens, all four of these questions are matters that style manuals do not answer with notable consistency or unanimity.
Mushy guidance
To illustrate the mushiness of style guidelines on these points, let me cite several relevant ones from The Chicago Manual of Style, fifteenth edition (2003). First, with regard to commas before quotations and questions:
6.53 Comma with quoted material. Quoted material, if brief, is usually introduced by a comma; if longer or more formal, by a colon [cross references omitted]. If a quotation is introduced by that, whether, or a similar conjunction, no comma is needed.
[Relevant example:] It was Emerson who wrote, "Blessed are those who have no talent."
...
6.55 Comma with questions. A direct question included within another sentence is usually preceded by a comma; it need not begin with a capital letter (see first two examples), but of the question is relatively long or has internal punctuation, an initial cap helps (see third example). ...
[First three examples:] Suddenly he asked himself, where am I headed?
The question, how are we going to tell her? was on everyone's mind.
Legislators had to confront the issue, Can the fund be used for the current emergency, or must it remain dedicated to its original purpose?
Second, with regard to capitalizing the first word of an embedded question (or not):
6.71 [Question Mark] Within a sentence. A question mark is used within a sentence at the end of a direct question (see also 6.55). If the question does not begin the sentence, it need not start with a capital letter.
Is it worth the risk? he wondered.
The question, how can the two be reconciled? was on everyone's mind.
Third, with regard to quotation marks (or not):
11.33 Quotations and "quotes within quotes." Quoted words, phrases, and sentences run into the text are enclosed in double quotation marks. ...
...
11.42 [Quotation Marks Omitted] Maxims, questions, and the like. Maxims, mottoes, rules, and other familiar expressions, sometimes enclosed in quotation marks, are discussed in 6.54 and 8.210. Questions that do not require quotation marks are discussed in 6.55 and 6.71.
Fourth, with regard to final punctuation appearing inside or outside the quotation marks, if quotation marks are used:
6.9 Colons, semicolons, question marks, and exclamation points. Unlike periods and commas, these all follow closing quotation marks unless a question mark or exclamation point belongs within the quoted matter. ...
[Relevant example:] Which of Shakespeare's characters said, "All the world's a stage"?
...
6.75 [Question Mark] With quotation marks, parentheses, or brackets. A question mark should be placed inside quotation marks, parentheses, or brackets only when it is part of the quoted or parenthetical matter. See also 6.9.
[Relevant examples:] Why was Farragut trembling when he said, "I'm here to open an inquiry"?
"What do you suppose he had in mind," inquired Newman, "when he said, 'You are all greater fools than I thought'?"
...
6.123 When to omit comma or period. Neither a period (aside from an abbreviating period) nor a comma ever accompanies a question mark or an exclamation point. The latter two marks, being stronger, take precedence over the first two. If a question mark and an exclamation point are both called for, only the mark more appropriate to the context should be retained.
[Relevant examples:] What did she mean when she said, "The foot now wears a different shoe"?
Who shouted, "Up the establishment!"
Vague conclusions
The various guidelines cited above indicate that Chicago supports the following conclusions regarding the four issues I identified at the beginning of this answer:
Chicago favors putting a comma before the beginning of the question, whether quotation marks are included or not. However, the fact that the modifier "usually" appears at crucial points in both 6.53 and 6.55 suggests that omitting the comma wouldn't be absolutely wrong.
Beginning the embedded question with a capital letter is not required, but it may be helpful in longer questions that have internal punctuation. This advisory wording again means that capitalization isn't wrong; rather, Chicago simply doesn't see it as helpful except in instances involving complicated questions.
Generally, direct quotations take quotation marks, but Chicago explicitly acknowledges an exception to this rule for questions—particularly unspoken questions (such as thoughts not voiced). The OP's question is hypothetical—not cited or attributed—so it might very well fall into the exception that Chicago carves out in 6.55 and 6.71, although we can't say for sure. In any case, Chicago doesn't seem to view quotations as being crucial to sense except in dialogue and in citations of things actually spoken or written. Also, Chicago endorses double quotation marks over single quotation marks—but that is a U.S. style preference and is at odds with the predominant style in the UK and elsewhere overseas.
Chicago seems to favor putting the question mark within the quotation marks in a situation where a sentence asks a question about an embedded quotation that itself ends with a question mark (where you might logically expect the sentence to end with ?"? to reflect the existence of both the quoted question and the question about it). Chicago definitely opposes double-question-mark punctuation, but the examples it provides show internal exclamation points overriding external question marks—not internal question marks doing so.
If you chose to omit the comma before the question and/or lowercase the first word of the question even though the question includes internal punctuation (the comma after "hey") and/or move the question mark outside the close quotation mark (if you elected to use quotation marks at all), it is not at all obvious that The Chicago Manual of Style would consider your decision(s) wrong. The upshot of all this is that you have many more options available, even under the guidelines that a single style manual provides, than the four possibilities you specify in the original question.
