Do I call their bluff if they are practically blackmailing me?

You made the choice to enter an affair with your employer. You knew it was wrong and yet you CHOSE to do what you did. If the choices YOU made cost your children a home and lively hood that is YOUR fault. The only way a person CAN blackmail you is if YOU give them something to use against you. Obviously you CHOSE to give this person something to use against you and now YOU must pay for your actions.