We are raycasting onto "npcs" in our game which is an actor to change mouse cursor types and behaviours. If you are close to the npc it doesn't cast correctly. but if you move away from it, it does. I have a video which shows clearly what happens:

http://www.youtube.com/watch?v=VFzdy9TNoaM

The code is as follows:

if(Ms.startPos() != Ms.pos()){T.obj =NULL;mst=MHand;
if(!Ms.hidden() || Gui.ms()==NULL)
{
Vec2 screen =(Ms.pos());
UInt actor_groups=~(IndexToFlag(AG_NPC));
Vec pos, dir; ScreenToPosDir(screen,pos,dir);
PhysHit phys_hit;
if(Physics.ray(pos, dir*D.viewRange(), &phys_hit, actor_groups))
{
T.obj =phys_hit.obj;
T.pos =phys_hit.plane.pos;
}
}
}
In NPC struct, if(Cur.obj == this)mst=MNpc;
Note mst is mouse types

In this line:
if(Physics.ray(pos, dir*D.viewRange(), &phys_hit, actor_groups))
dir looks like it's relative to pos. So maybe something like pos+dir*D.viewRange() could work?

@scotty1121:
UInt actor_groups=~(IndexToFlag(AG_NPC));
specifies everything except NPC
please use debugging to investigate results of the raycasting

@Rubeus: no, actually the 2nd param is the delta, so the codes for that are ok from scotty1121