You want the compass to act like a compass right? I had something similar to this. I was making a view mode were i wanted the player to turn to the acute angle of the camera and this is what i came up with:
Code:
//This is called when a movement key is pressed
ScreenToPosDir(Vec2(0,0), tpos, screendir)
angle.set(tpcamera(screendir.xz(),angle.x),angle.y);
//The turning function
Flt tpcamera(Vec2 &screen, Flt chrangle)
{
Flt chrangletemp =AngleFull(chrangle);
Flt screenangletemp =AngleFull(Angle(screen)-PI_2);
Flt offset=0.0;
if(screenangletemp>=PI)
{
if(chrangletemp-0.04>screenangletemp||chrangletemp<(screenangletemp-PI))
offset=-0.03;
if(chrangletemp>(screenangletemp-PI)&&chrangletemp<screenangletemp)
offset=0.03;
}
if(screenangletemp<=PI)
{
if(chrangletemp+0.04<screenangletemp||chrangletemp>(screenangletemp+PI))
offset=0.03;
if(chrangletemp<(screenangletemp+PI)&&chrangletemp>screenangletemp)
offset=-0.03;
}
return chrangle+offset;
}
chrangletemp is offset by 0.04 to prevent jitters.
Replace screenangletemp with 0/2pi (i think that should be "north"..) and draw the line pointing to north based on the return value of the function. You can then calculate the other lines (south,east,west) based on that value (+/- 90 degrees).
For the up and down thing maybe this will work. I was thinking to make the character look up/down based on the y direction of the screen like this : angle.set(tpcamera(screendir.xz(),angle.x),tpcamera(angle.y,screendir.y()); so maybe you can use the return value of tpcamera(angle.y,screendir.y()) for your compass too (replace screendir.y() with something else).
Well, times up; gotta sleep. This is an interesting problem and i'll be here tomorrow
hope i helped