As someone who spends a lot of time writing software to control railway signals, the usual term is throw a set of points.
Alternative terms are specific to the direction:
Pull the points moves them such that the train moves off the main line ("Reverse" position).
Push the points moves them back to the straight line position ("Normal" position).
Push and Pull are used as, by convention, this is the direction in which the lever is moved when it is located in a signal box, as opposed to trackside.
To be pedantic, if the driver is getting out of the cab to manually throw a lever, then it is a ground frame.
In the US (and the London Underground), points are known as switches.