Figure labels should be capitalized, but references to them should not, according to the Chicago Manual of Style (CMOS).
The Chicago Manual of Style Online (CMOSO) states that "In text, the word figure is typically set roman, lowercased, and spelled out except in parenthetical references ('fig. 10')," as in "'as figure 1 shows . . . ,' 'compare figures 4 and 5'" (3.9).
I was unable to find anything in the CMOSO directly addressing the question of whether to capitalize the figure label, but section 3.23 offers this example: "Figure 3. Detailed stratigraphy and geochronology of the Dubawnt Supergroup." Additionally, my hard copy of the 14th edition of the CMOS specifies that labels should be capitalized.
Different style guides will probably have different things to say about this though. Consistency is the important thing; if you have an in-house style guide, consult that. (If you don't, consider creating or adopting one!)