I used to trade bonds (amongst other stuff) so I can tell you how it works for bonds. Presumably it's similar for stocks:
As you know, going short means you sell the bond/stock. Generally going short implies you are going negative, as opposed to selling inventory you already own. So if I go short 100 million 10y bunds (German government debt) this means I sell 100m bonds I don't have.
The guy I am selling to doesn't know I am going short, he just knows I've sold him the bonds and he expects delivery at the end of the day (or possibly in 1 or 2 days time, depending on the definition of "spot" and various other things).
So I have to deliver him 100m bonds I don't have. Where do these come from? This is where the "repo" or repurchase market comes in to play. Very simply, the repo market is short-term buy/sell market: Party A agrees to sell Party B some bond at some price, AND additionally agrees to buy it back a short while later for a slightly higher price. What Party A has effectively done is borrow money from B and put up the bond they own as collatoral.
So what happens after I short my bond is that I go to the repo traders and say "hey - I'm short 100m bunds, you need to do a repo trade to flatten my position". So these guys will loan out some cash (around 100m euros) and take as collateral 100m bunds. These I use to deliver to my seller.
Edit: In reality I don't tell the repo traders I've gone short a bond. It's their responsibility to make sure all positions are repo'd out. They see a netted view of all bond positions across the bank and repo just the net positions.