Don't forget that there were three with the name James - the letter of James was written by James the brother of Jesus, who took over and became the head of the original 'church' (if you can yet call it that) after the 'empty tomb' in Jerusalem - so he was an Aramaic Jew writing for mainly Jews in Jerusalem, (he was later murdered in Jerusalem at the order of the high priest during a short gap between Roman governors).
He was called 'James the Just' and was the one of which it was written "James was the brother of Christ according to the flesh" - he was not the James of spain.
Then there was James, son of Alphaeus, one of the disciples, and of whom almost nothing is known (linking with James the Just is erroneus and was done as a method of divorcing Jesus from any possible brothers and sisters).
'Our' James, James the Greater, was the first to be marytred (Agrippa 1 and by the sword). This James of Spain was James the Greater, son of Zebedee, one of the 'sons of thunder' with his brother John, and the first disciple of Jesus .. so the singular appearance recorded by Paul was most probably to the brother of Jesus, James, as the disciples have already been mentioned. Though it could have been to another James of whom we know nothing and who has since fallen through the net of history.
One of the problems of the Iberian mission before his return to Jerusalem and murder is that there are no early church references to it ... there are references to Thomas having a mission to India and so on but not James to Spain. This does not mean he didn't do it- absence of evidence isn't evidence of absence after all. I'm not sure but I think the first refrences are from the 7th century ..
(thinks: is this not history?)