This is the (literal) textbook definition of a security:
any note, stock, treasury stock, security future, bond, debenture,
evidence of indebtedness, certificate of interest or participation in
any profit-sharing agreement, collateral-trust certificate,
preorganization certificate or subscription, transferable share,
investment contract, voting-trust certificate, certificate of deposit
for a security, fractional undivided interest in oil, gas, or other
mineral rights, any put, call, straddle, option, or privilege on any
security (including a certificate of deposit) or on any group or index
of securities (including any interest therein or based on the value
thereof), or any put, call, straddle, option, or privilege entered into
on a national securities exchange relating to foreign currency, or, in
general, any interest or instrument commonly known as a “security,” or
any certificate of interest or participation in, temporary or interim
certificate for, receipt for, guarantee of, or warrant or right to
subscribe to or purchase, any of the foregoing. [1]
It's a somewhat circular definition, but my reading is that basically, anything whatsoever that gives you an ownership interest or claim on the future profits of an enterprise can be construed as a security. You can track who owns the shares of the enterprise via entries in a central ledger (the current method for most stock companies), use paper bearer certificates (traditional way), use a distributed ledger, hand out carved pieces of pottery, whatever. If it functions even remotely like a security, it's probably a security.
EDIT: The Supreme Court's "Howey Test" [2], which came out of a 1946 case over complex real-estate leaseback deals, is in some respects a simpler method to determine whether something may be a investment contract, which is a security:
1. It is an investment of money
2. There is an expectation of profits from the investment
3. The investment of money is in a common enterprise
4. Any profit comes from the efforts of a promoter or third party
The Howey Test is reliable in the sense that anything satisfying those four points is extremely likely to be an investment contract and thus a security, but I believe there are probably securities that fail one or more aspects of the test but are still regulated, due to the broad statutory definition of "security" under the 1934 and 1940 Acts.
[1] That's from the Investment Company Act of 1940, which seems to be definitional; there are some other definitions used elsewhere in the UCC and other Federal laws (largely from 1934), but they seem to be similar, and quite a few places punt to this definition. (Quoted in http://apps.americanbar.org/buslaw/newsletter/0014/materials... which is an annoyingly "locked" PDF that you will need to de-DRM; this is left as an exercise to the reader.)
[2] http://consumer.findlaw.com/securities-law/what-is-the-howey...