It is the employer who is primarily at risk (I generally agree with rjurney's related comment).
The traditional assessment (i.e., one done without regard to any safe harbor protections and also without going mindlessly through the "20 factors," which is usually not helpful) would run roughly as follows: (1) do you have a written agreement specifying that you are an independent contractor and setting forth terms that characterize an independent contractor relationship? (if you don't have such an agreement, the company basically loses out the gate on this issue in almost all cases); (2) are your services of the highly skilled type that people normally would pay an outside contractor to perform? (if so, this tips toward contractor); (3) are you truly in business for yourself and do the various indicia (yes, that is a lawyer word) of your work situation corroborate this (if sole proprietor, do you have a business license, a dba, a policy of liability insurance, etc. relating to your contract work or are you set up as an entity - these factors all tip toward contractor); (4) can you document that you perform services for multiple clients, that you hold yourself as doing so, and that your agreement permits work for others (if you have a website advertising your company, or if you in fact have done contracting work for multiple clients over a sustained period, etc., this tips toward contractor); (5) is what you do for your company a regular and recurring part of its business, for which it would normally hire an employee as opposed to more of a spot project such as helping develop a company's website? ("regular and recurring" tips strongly toward employee; "spot" tips toward contractor); (6) do you have to report to a specific place of work on a specific schedule? (tips strongly toward employee); (7) do you work under the direct supervision of any person at the company, whether a boss or a supervisor? (tips strongly toward employee); (8) are you required to complete a clearly defined "project" or must you show up at work to do whatever you are ordered? (the latter tips strongly toward employee).
All of the above is another way of saying, "is this person really in his own business?" when performing duties for this company. If I am hired as the company's lawyer, my goal is to deliver a result on my own time and in my own way. If what you are doing is analogous to this, then you might truly be a contractor. If not, then you likely are not.
Of course, check with a knowledgeable lawyer about your own specific case and concerning any risk or options should you be mis-classified. Can't give advice here about any specifics.