I can't speak personally of Adderall and anxiety as I got on Dex right away and since it works I never left.
Dexedrine made me feel a little "edgy" at first but soon (less than 3 months) I felt LESS anxiety than any time in my adult life. It was very amazing to me as I worried endlessly about all sorts of insignificant things.
I think it is a combination of the right med and time. All amphetamines can make adrenaline (norepinephrine) surge and that causes a biological "fight or flight" in many people. That is normal and goes away.
The best thing is if it's only a little anxiety you feel - give it a couple of months and keep track of how you feel.
bman,