Why is castling out of check not allowed?

The Laws treat the three king squares the same way. Art. 3.8 says castling is prevented while the square the king stands on, the square it crosses or the square it lands on is attacked. Being in check is simply the first of the three, so there is nothing special going on: the king may not be under attack at any point in the move, and that includes before it starts.

Does being checked cost me the right to castle?

Only if you answer it with a king move. A check does not touch your castling rights by itself. Capture the checking piece, or block the line, and your king is still on e1 with both rooks at home, so both castles are still there. That is a good reason to look for a block before you reach for the king.

Can you castle if the rook is attacked rather than the king?

Yes, that is a different question and the answer is the opposite. The rule only cares about the king's three squares. A rook on h1 or a1 sitting under attack from a bishop or a queen does not stop the move at all, and after castling it is usually defended by the king anyway.

Can you castle out of check?

A knight lands on d3 with check, and O-O looks like the perfect answer: the king runs into the corner, the rook covers f1, and the knight is left hitting nothing. It is not legal, and an arbiter will make you take it back.

The knight on d3 checks the king on e1. White has exactly four legal moves: cxd3, Ke2, Kf1 and Kd1. Castling is not among them.

Deal with the check first

The wording in the FIDE Laws (Art. 3.8) puts the square the king stands on in the same sentence as the square it crosses and the square it occupies. All three must be free of attack. So being in check kills castling for that turn in exactly the way an enemy bishop hitting f1 would.

In the diagram White has four moves and only one of them is any good. 1.cxd3 takes the knight with the c-pawn and answers the check by removing the piece that gave it. The king has not moved. Both rooks are at home. So 2.O-O is available the very next turn, and White has won a knight into the bargain.

The three alternatives, 1.Ke2, 1.Kf1 and 1.Kd1, all solve the immediate problem and all cost the right to castle forever. That is a real price. A king shuffling on d1 or f1 with the rooks still stuck in the corners is a position I would rather not defend.

The right outlives the check

This is the part people get wrong. A check does not remove your castling rights. Only moving the king does that, or moving the rook.

So the order of operations at the board is: can I capture the checking piece? If not, can I block the line? If neither, the king has to move and I have lost the castle. Look for the capture and the block first, every single time, and you will keep the castle far more often than the players who grab the king by reflex. The same three answers are the whole of how you get out of check, and the castling question is just the choice between them.

Castling as an escape, one move later

There is a version of the idea that does work, and it is worth knowing because it wins games. If a check can be blocked cheaply, block it, and then castle into the shelter you were aiming for anyway. Your king ends up on g1 behind f2, g2 and h2, which is exactly where you wanted it, and the opponent has spent a tempo on a check that achieved nothing.

What you cannot do is skip the middle step. The move O-O has to be legal on the turn you play it, and while your king is under attack it is not. If you are not sure whether a position allows it, the full list of conditions is short enough to memorise, and a handful of castling puzzles will make the check-first habit automatic faster than reading about it will.