Using a mouse to select objects in 3D is a little tricky because the mouse gives only 2D pixel coordinates which must be some how converted to 3D coordinates. In fact, the mouse location on screen represents an the infinite number of points in world space which are projected on to a single point in screen space.
In a 3D envorinment, there may be more than one object under the mouse pointer when it is clicked. Normally, the user's intention is to select the object which is visible at this point.
The general approach will be to use the mouse coordinates to generate corresponding points on the near-plane and far-plane in world coordinates. These points will form a ray. The ray will be compared against every object (or its bounding volume) for intersection. If more than one object is intersected, the object nearest the viewer is selected.
Window Coordinates (pixel Coordinates) usually have the origin in the upper left hand corner, the height and width are measured in pixels.
Screen space is the coordinate system generated after projection. In screen space, the center of the window is (0,0), the top of the window is at y=1, bottom at y=-1, and the width is defined by the aspect ratio (left=-aspect, right = +aspect).
screen_space_x= ((( mouse_x/width)*2)-1.0)* aspect; screen_space_y= 1 - ( mouse_y/height)*2;
In screen space, the far plane is at z=1, the near plane is at z=0, so the start and end points of the ray under the mouse is;
near={ screen_space_x, screen_sapce_y,0};
far={ screen_space_x, screen_sapce_y,1};
The near and far planes are used as terminators of the ray, because objects outside this range are invisible to the user and therefore the user would not attempt to select them.
The two end points of our ray need to be converted to world coordinates by applying the inverse of the view and projection matrices.
To test for intersection of an object, the ray needs to be compared with the bounding volume of the object. This would nessesitate applying the object's world transforms to the bounding volumes.
It is usually more efficient to apply the inverse of the world transform to the near and far points.
near'=near * (World*View*Proj)-1; // convert to object space
far'=far * (World*View*Proj)-1;
Now it is a matter of testing the ray(near', far') for intersection against the bounding volume(s).
If the ray intersects AND the intersection time is between 0 & 1, then this object lies under the mouse.
Check these notes for intersection techniques
When several objects are intersected, we select the object with the smallest intersection time (but greater than zero.
XNA provides a number of useful functions and classes to help with object picking;
This project demonstrates Object picking in XNA.
At the moment(December 2006) XNA (Beta2), does not have support for displaying a mouse pointer, so this is simulated in the demo using a sprite, whose position is controlled by the mouseState object.