Understanding here depends less on the meanings of church than on the meanings of go.

There are numerous uses of go. Most commonly it refers to moving or traveling somewhere. In this sense, and when by church we mean a building used for Christian worship, we use the article with church according to the usual rules:

I'm sure I lost my camera in Montmartre; I went to a church there— but I don't recall which one— and left it in a pew.

 

We stopped for lunch in a small town, and I walked around a bit after we ate. I went to the church, then the square, then got an ice cream soda at the drugstore.

Go can also mean to attend or visit a place or type of place for a particular purpose, however. To say you go to church means not only that you physically situate yourself at the building, but that you are engaged in regular worship services there. In this sense, you do not use an article.

I went to church in the morning so I could watch the football game later.

 

I went to church growing up, first Blessed Sacrament and then St. Ann's after we moved. But I lapsed when I moved to the city.

The same change of meaning applies for a number of other words which can denote both a location and a particular engagement: court, school, market, town, and so on. To go to a jail is to visit a penitentiary facility; to go to jail is to be incarcerated; don't mix up the two in conversation.

English being English, unfortunately, this is not a strict rule. Most geologic features, for example, require a definitive article when used in either a specific or generic sense: the mountains, the shore, the woods, etc. Certain proper nouns always take the definitive article as well. Thus, a simple statement can be ambiguous.

I went to the Church of St. Luke when I lived in Lexington.

could mean that you once visited the building known as St. Luke's, but it could also mean that you were a regular parishioner who attended services every Sunday.

I went to the beach last summer.

could mean you visited a particular beach once last summer, but it could also mean you went to one or a number of different beaches as a regular activity last summer.

As always, context is key.

Understanding here depends less on the meanings of church than on the meanings of go.

There are numerous uses of go. Most commonly it refers to moving or traveling somewhere. In this sense, and when by church we mean a building used for Christian worship, we use the article with church according to the usual rules:

I'm sure I lost my camera in Montmartre; I went to a church there— but I don't recall which one— and left it in a pew.

We stopped for lunch in a small town, and I walked around a bit after we ate. I went to the church, then the square, then got an ice cream soda at the drugstore.

Go can also mean to attend or visit a place or type of place for a particular purpose, however. To say you go to church means not only that you physically situate yourself at the building, but that you are engaged in regular worship services there. In this sense, you do not use an article.

I went to church in the morning so I could watch the football game later.

I went to church growing up, first Blessed Sacrament and then St. Ann's after we moved. But I lapsed when I moved to the city.

The same change of meaning applies for a number of other words which can denote both a location and a particular engagement: court, school, market, town, and so on. To go to a jail is to visit a penitentiary facility; to go to jail is to be incarcerated; don't mix up the two in conversation.

English being English, unfortunately, this is not a strict rule. Most geologic features, for example, require a definitive article when used in either a specific or generic sense: the mountains, the shore, the woods, etc. Certain proper nouns always take the definitive article as well. Thus, a simple statement can be ambiguous.

I went to the Church of St. Luke when I lived in Lexington.

could mean that you once visited the building known as St. Luke's, but it could also mean that you were a regular parishioner who attended services every Sunday.

I went to the beach last summer.

could mean you visited a particular beach once last summer, but it could also mean you went to one or a number of different beaches as a regular activity last summer.

As always, context is key.