Such is an adjective in both cases, which can just be likened to "like this," for me.
Such fun = fun like this/fun of this sort Such a quiet girl = a quiet girl like this/a quiet girl of this sort
The first such and the second are the same, though the first intensifies the noun because of the lack of an article to separate the noun from its modifier.
Whether it's a determiner or not depends on the context.
"Such fun" as a determiner would imply the speaker is referring to a any particular kind of fun that has been stated. The sentence, if meant to have such used as a determiner, would be better reworded ("any such fun").
But if the sentence is just intensifying, as it is, it's acting as an adjective.
In the two respective sentences in the question, it's the difference between the adjective being: "A quiet girl like that (comparison between the girl and a similar but unknown noun)/A great deal of fun" and the determiner being: "Such quiet girls/Any fun".
The determiner doesn't really make sense in these cases.
