I think that's (way) too strong.
"Hi, I'm Josh!" refers to me as the agent of action,
even if all it does is stick a label on me.
There are all kinds of indexical statements that would
seem nonproblematic, although their apparent reference is to
some abstraction. "I like pretty flowers".
It's not the agency, or even the experiencer,
it's a failure to ground these kinds of terms in physicalism.
(or, if you're an orthodox Wittgensteinian, failure to in some
other way ground the terms, or their particular usages, in your
favorite methodology)
Thus we recognize the value of these "folk" terms,
without being misled by their reification.
