Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres • Hacker News Board

Re: Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres

#31

Post by joshdick » Wed, Apr 06, 2016, 7:42 PM UTC

npace12 wrote: post #30

sperand_io wrote: post #10

Earlier quoted context omitted.

Great question! When you enable a source, we begin running a job on an interval for you that pulls the data from the source, applies some light normalizations and transformations, and sends the data to our Object API (which is in charge of upserting the data into Segment and flushing it to your warehouse). In Salesforce's case, we issue bulk queries to pull the complete collection on the first run, then modify the qu…

You should make this available for trial. No way I'm paying $449 just to try it. I set it up with the developer account, but when I try to connect the integration, it says it's not available for the developer plan. I was like "ok, fine, i'll pay 50 bucks to try it" but then it says salesforce is only available on the Growth plan, which is $449. Closed the tab, and moved on with my life.

This product has Salesforce replication, and it's free to try: https://rjmetrics.com/product/pipeline/

Re: Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres

#32

Post by n2parko » Wed, Apr 06, 2016, 8:22 PM UTC

slachterman wrote: post #27

Are there any plans for Warehouses support for IBM's DashDB?

Hey slachterman DashDB isn't yet on the roadmap, mind filling this out so we can follow up?

https://segment.com/contact/requests/warehouse

Re: Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres

#33

Post by n2parko » Wed, Apr 06, 2016, 8:22 PM UTC

vyrotek wrote: post #25

Please add options to use Windows Azure as a warehouse!

hey vyrotek thanks for the request! mind filling this out a request and we'll follow up? https://segment.com/contact/requests/warehouse

Re: Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres

#34

Post by sperand_io » Wed, Apr 06, 2016, 8:53 PM UTC

npace12 wrote: post #30

sperand_io wrote: post #10

Earlier quoted context omitted.

Great question! When you enable a source, we begin running a job on an interval for you that pulls the data from the source, applies some light normalizations and transformations, and sends the data to our Object API (which is in charge of upserting the data into Segment and flushing it to your warehouse). In Salesforce's case, we issue bulk queries to pull the complete collection on the first run, then modify the qu…

You should make this available for trial. No way I'm paying $449 just to try it. I set it up with the developer account, but when I try to connect the integration, it says it's not available for the developer plan. I was like "ok, fine, i'll pay 50 bucks to try it" but then it says salesforce is only available on the Growth plan, which is $449. Closed the tab, and moved on with my life.

Sorry for the confusion there — our integration that sends data to Salesforce is on the integrations growth tier, but the pricing for data sources and warehouses is separate (and does indeed have a free trial :).

Your point prompted us to add a callout in the UI to prevent this sort of confusion going forward (see here: https://www.dropbox.com/s/yoc6a3nvsoupesa/Screenshot%202016-...)... and more generally we'll be working to make the distinction between data sources and data destinations clearer from a UX perspective in the near future.

Re: Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres

#35

Post by georgewfraser » Wed, Apr 06, 2016, 11:20 PM UTC

samcheng wrote: post #8

ETL-as-a-Service is a great idea, particularly one that is visualization/analytics-tool-agnostic! However, there are so many data sources, and they all require different integrations with their different APIs or export mechanisms. A service isn't really useful unless it can import the lion's share of services that a given company uses...

My company (Fivetran, YC W2013) offers the same service and supports a lot more sources (https://fivetran.com/integrate), including relational databases.

Re: Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres

#36

Post by grinich » Wed, Apr 06, 2016, 11:29 PM UTC

josep2 wrote: post #17

Segment is always killing it.

They're totally doing great! (But we should use less violent language to describe it...)

Re: Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres

#37

Post by greggyb » Thu, Apr 07, 2016, 12:16 AM UTC

dwmintz wrote: post #24

greggyb wrote: post #15

Earlier quoted context omitted.

From a BI consultancy perspective, not in the slightest. The amount of time we spend on ETL is not because of the difficulty of piping data around from place-to-place. The difficulty is in modelling data appropriately to support ad-hoc analyses. Rather, the E and L portions don't provide much difficulty (hassle, frustration, sometimes time, sure, but they're not inherently difficult). The T, transformation, is huge i…

Agree that it's only one part of the data lifecycle, but if you don't have your own team of engineers to build all those connectors, you can't even get to the point of being able to transform your data. We at Looker (disclosure: work there), are totally focused on making the transformations fast, flexible and powerful. But without the data to transform, there's nothing we can do to help customers. So we're super psyc…

Absolutely. I was just trying to illustrate where a single tool doesn't pose a threat to BI companies, particularly from a consulting angle.

Tools like Segment and yours are invaluable. I work for a Microsoft partner, so often am stuck in that ecosystem for better or worse, but the problems and solutions in the BI space are largely universal and transcend specific tools. Even when the tools make individual tasks trivial, the overall architecture and design of a data pipeline and visualization solution leave plenty of room for companies like the one I work for.

Re: Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres

#38

Post by pinaceae » Thu, Apr 07, 2016, 1:21 AM UTC

just be careful with SFDC, bunch of harsh limits, killer one being 5k bulk api requests per rolling 24h.

if you think that's a lot you haven't seen big orgs with a shit ton of integrations and custom stuff on top of them.

Re: Segment Sources – Load Salesforce, Zendesk, Stripe into Redshift and Postgres

#39

Post by sandGorgon » Thu, Apr 07, 2016, 2:09 AM UTC

pkrein wrote: post #7

ngould wrote: post #6

Earlier quoted context omitted.

Hey pkrein, I am a Segment fan and startup-scale user of your warehouse service. The warehouse service is super convenient, but its main shortfall for us is that you don't offer historical backfilling of data more than 60 days before the warehouse was enabled. That is the data we need in our warehouse most. There are other ways to load in Salesforce, Zendesk, or Stripe data. It is certainly nice to be able to do that…

Yep, as part of this launch we've revised our pricing. If you commit to an annual plan (even self-service plan) we will load all of your historical data, not just 60 days. Just email friends@segment.com after signing up for the paid annual plan and we'll make it happen on the next sync.

Interesting. The pricing is a little confusing for the rest of us who are considering Segment for the first time (the warehouse part..not the integrations) - especially the size of data and how we load it.

Maybe you are estimating data size based on known types of data in salesforce,zendesk,etc. But what if i want to load data from my internal dB as well? The 60 day notions,etc.