Maybe not human flesh but she has been "tinkering" with them the whole time so that is why the pitchfork affected them.

the nanny has been secretly feeding them human flesh.
You can't persuade fanboys. You'd be better off trying to convince a wall. ~CodeNamePlasmaSnake~

Im pretty sure the girls have to be actively siphoning for that to work if not Caroline wouldn't be able to even touch the girls for too long. Plus they would've drained magic from the pitchfork if it was a subconscious thing. I'm pretty sure the writers just hope no one will catch the many plotholes they keep digging.

If it was because of magic, Stefan should be bothered by it too. We know siphons can siphon the magic from vampires.
What the other poster said; the twins siphoning off Selena makes good sense. What I do not understand is how Alaric did not make that link and believes Georgie is the siphon over his babysitter who actually spends a lot of time around the girls.

Also why is Serine interested in the twins, what are her plans for them? So many questions! Haha.
It's a jungle, I'm a freak. Hear me talk, but never speak.

Another theory, is it possible the girls have been unknowingly siphoning off Serine's siren powers? Which might make them sensitive to the pitchfork sound as well?
It's a jungle, I'm a freak. Hear me talk, but never speak.

I thought maybe it just had to do with their magic and the reason it didn't hurt bonnie is because she doesn't have hers (again,but that's another thread). I don't know if being a siren is just a psychic thing or magic but I can't recall any other physics on the show it was always some form of magic.