I've been wearing FiveFingers (FFs) regularly since 2007. I have owned over 20 pairs (various Sprints, Surge, KSOs, KSO Treks, Speeds, Spyridons, and Lontras) and unless its winter or really rainy and I'm not hiking, FFs are my daily footwear.
For me, FFs changed my life. I was pretty rough on my knees growing up and had struggled with knee pain since my teens. So when I tried FFs in my 20s, they were a game-changer. I could scramble up mountains again and not be crippled during the descent and for the following week. Being young, I was able to transition to wearing them full-time, slowly building up my foot muscles; arch/ankle support isn't an issue if your arches/ankles are strong. Not everyone can do this, though.
Since then I've done multi-day backpacking trips over very rugged terrain carrying a very heavy pack (35lbs+ - cold, volcanic wasteland areas where you have to pack in ALL your water). Tried it ONCE with boots, thinking my FFs wouldn't be enough - bad idea. I was in so much pain that night I didn't sleep. Now I backpack exclusively in FFs. The only terrain I can't handle well in them is asphalt, which has me worried a bit about my future Camino in 2018. :\
As far as socks go, unless I'm using them as rock shoes while swimming, I wear Injinji socks with FFs. I mostly wear socks because it prevents the shoes from getting stinky.
The ONLY pair of mine I've ever gotten blisters in are the 2007 Surges - it was an early model with poorly planned seams, otherwise I never get blisters and this includes a multi-day trip where where I went 26 miles in one day, up and down mountains, over rocky terrain, with a 40lb load with holes worn in the sides of the toes and volcanic ash (abrasive) kept getting in, plus I had to ford a few rivers, so I had wet feet from time to time.
But again, FFs happen to be for me and I have spent years in them. I think they really do take a lifestyle change to be beneficial because it doesn't take long for your foot muscles to atrophy if you aren't using them constantly. Also the fit is SO different across all the different models; it can be a bit of a Goldilocks hunt to find just the right pair for you, and sometimes that isn't evident until you've worn them a while.