Let's capitalize the first word of the sentence and assume that "pray" is a typo for some adjective that properly modifies "house," perhaps "prayer house" meaning a place of worship.
Your first version places the prepositional phrase "with hardly any walls" immediately after "straw," which may mislead your readers into initially parsing the building material as straw that has few walls. That's impossible, so readers will be quick to realize that it's the house that has few walls.
Thus your second version is slightly kinder to your readers by placing the house modifier next to "house." Notice that the comma after walls saves the "built" phrase from the same fate -- it would apply (restrictively) to the nearby "walls" only in the absence of the comma.
There are some semantic oddities -- the use of "pray," the verb set, and the use of "little" and "hardly any" -- but those aren't grammatical issues.
Be aware that it's hard to understand the description of walls without visualizing the following image:
If that's not what you have in mind, you might reconsider your description.