Depending on whether the prisoners are able to determine the passage of time, this can either take relatively little time or a lot of time.
Easy: If the warden grabs a prisoner once a night, then the prisoners set up a system according to the passage of days.
Each prisoner starts with four "tokens." This means that in a prison population of 23 prisoners, there are 92 tokens in total for the prisoners.
We'll call the switches A and B. Starting off, A signifies 1 token. When you turn the switch on, you are putting a token into the switch. When you turn it off, you are taking one. B signifies 2 tokens.
When a prisoner visits the office, he looks to see if he can grab some tokens. If neither are on, then he puts in some tokens of his own. He will try to put as many tokens as possible in. So, for example, if he has 4 tokens, he will put in two.
Of course, if he has insufficient tokens, he does nothing.
After a predetermined period of time, the switches double in "worth." Switch A is now worth 2 tokens, and Switch B is worth 4. This will double again to 8 and 4, then 16 and 8, and so on, until they reach 64 and 32. If someone is able to accumulate 92 tokens, then it's apparent that everyone has visited the room at least once. Otherwise, it then starts over at 1 and 2.
We need four tokens for every prisoner because of a few possible extra tokens. If A and B are both on, then there are three extra tokens in the system. If you have fewer than four tokens per prisoner, it becomes possible to accumulate the required number without having everyone be in the room. You also can't have fewer tokens, because it would require that a prisoner collect tokens that might not actually be there.
Hard: If no one can figure out the passage of time, then they have to stay at 1 and 2. This will take much longer for someone to eventually accumulate all of the tokens, especially since all of them are trying to accumulate.
Edit: Here's a horribly written Python program that shows this process in action: http://codepad.org/iY121Ui3