Yes, French water fountains marked as either ‘eau potable’ or simply ‘potable’ have water which is safe for drinking. We came across a great website that lists water fountains for drinking in France.
Something that I have learned living in France is that cemeteries are a great place to find drinking water if you can’t locate it anywhere else. They will generally have a tap for people to water gardens and for flowers.