Granite Falls is located near the southern border of Caldwell County at 35°48′1″N 81°25′59″W (35.800345, -81.432929). It is bordered to the south by the town of Rhodhiss, to the southwest by Rhodhiss Lake on the Catawba River (Lake Hickory), and to the west by the town of Sawmills. The center of town is located on a ridge between the Catawba River to the west and Gunpowder Creek, a tributary of the Catawba, to the east.
U.S. Route 321 (Hickory Boulevard) is a four-lane highway that passes through the east side of town; it leads 6 miles (10 km) southeast to the center of Hickory and 10 miles (16 km) northwest to Lenoir, the Caldwell County seat. US 321 Alternate runs through the center of Granite Falls as North and South Main Street.
According to the United States Census Bureau, the town has a total area of 5.3 square miles (13.6 km2), of which 5.2 square miles (13.5 km2) is land and 0.04 square miles (0.1 km2), or 0.84%, is water.
As of the census of 2000, there were 4,612 people, 1,758 households, and 1,211 families residing in the town. The population density was 1,080.4 people per square mile (417.0/km²). There were 1,849 housing units at an average density of 433.1 per square mile (167.2/km²). The racial makeup of the town was 92.06% White, 2.34% African American, 0.17% Native American, 0.39% Asian, 0.07% Pacific Islander, 3.95% from other races, and 1.02% from two or more races. Hispanic or Latino of any race were 6.07% of the population.
There were 1,758 households out of which 33.9% had children under the age of 18 living with them, 52.3% were married couples living together, 12.9% had a female householder with no husband present, and 31.1% were non-families. 25.4% of all households were made up of individuals and 11.8% had someone living alone who was 65 years of age or older. The average household size was 2.53 and the average family size was 3.01.
In the town the population was spread out with 24.4% under the age of 18, 8.8% from 18 to 24, 31.1% from 25 to 44, 20.7% from 45 to 64, and 15.0% who were 65 years of age or older. The median age was 36 years. For every 100 females there were 89.4 males. For every 100 females age 18 and over, there were 84.9 males.
The median income for a household in the town was $38,596, and the median income for a family was $47,064. Males had a median income of $28,309 versus $21,374 for females. The per capita income for the town was $17,750. About 5.7% of families and 8.2% of the population were below the poverty line, including 10.6% of those under age 18 and 9.7% of those age 65 or over.