If you want something simple without writing much code, go with PayPal. You'd expect some "horror stories" from a payment processor with 232 million accounts. Virtually all of them involve someone doing something that'd raise flags at any merchant account provider or bank, it's just that people think PayPal isn't a bank so they should be able to get away with anything. 10,000 horror stories still leaves 99.99% of users happy.
If you want to accept credit cards on your own site, you need to learn about PCIDSS compliance, you need to apply for a merchant account, and you need to a payment gateway that works with that merchant account.
It's well within reach of a lean startup. The application is usually a page or two long, you'll get approval from the underwriting bank within a week or so, and most MAPs also resell the payment gateway accounts so they set that up for you and you get everything at once. The fixed costs will be a statement fee and a gateway fee, $20-30 a month total.
It's terribly difficult to understand the fees you'll be paying with a merchant account. They're far more complex than whatever you get quoted will lead you to expect, and it's absolutely common for providers to change the fees on a monthly basis such that after a year you have no idea what you're paying. There can be dozens of classes of credit cards each with different rates when you charge them. My only advice for dealing with that is to try not to tie your code to your processor so that, when you grow enough for the fees to matter, you can shop around for a better deal.
You don't get to take shortcuts with the PCI stuff. The fine for a credit card being compromised from your server because you weren't PCIDSS compliant start at $500,000 per incident (paid to Visa/MC) plus legal fees and costs to provide credit monitoring to the victims.
If you accept credit cards directly on your site, you'll be required to fill out a self-assessment questionnaire and have a compliance scan against your server run on some basis. For most ecommerce setups, it'll be quarterly. What that costs depends on your merchant account provider; some contract with and pay a compliance company for you, others you have to pay some or all of the costs yourself, which can be a couple hundred dollars a year. SecurityMetrics seems to be the most popular service for the scans.
The technical part is easy. An Authorize.net integration will take you a couple hours at most. There are libraries for the popular payment gateways in every programming language known to man, and prebuilt shopping carts will have plugins/modules/whatever written by people already.
It's just like any other API. You POST some data to some server to make a charge/authorization/refund/etc and get a response to parse. If your app integrates with Twitter, you've already done much harder work than integrating a form with a payment gateway.