I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…
Thanks for sharing your experience. I work with Pachyderm, which is an open source data pipelining and data versioning framework. Some things like might be relevant to this conversation are the fact that Pachyderm is language agnostic and that it keeps analyses in sync with data (because it triggers off of commits to data versioning). This makes it distinct from Airflow or Luigi, for example.
Dgsh – Directed graph shell
41–50 of 53 posts
Re: Dgsh – Directed graph shell
#42I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…
We've been working on a directed graph execution engine called Converge https://github.com/asteris-llc/converge . In this case the task resource http://converge.aster.is/0.5.0/resources/task/ might help, as it allows you to create a directed graph using any kind of interpreter (for example, Python or Ruby) instead of having to use the DSL.
Re: Dgsh – Directed graph shell
#43Earlier quoted context omitted.
We've been working on a directed graph execution engine called Converge https://github.com/asteris-llc/converge . In this case the task resource http://converge.aster.is/0.5.0/resources/task/ might help, as it allows you to create a directed graph using any kind of interpreter (for example, Python or Ruby) instead of having to use the DSL.
You call this as a configuration management tools on Github. Does that make this a competitor to Ansible, etc as well?
Re: Dgsh – Directed graph shell
#44I am not familiar with the project. What are the advantages of Dgsh in comparision to pipexec: https://github.com/flonatel/pipexec
Re: Dgsh – Directed graph shell
#45This looks like potentially a great tool. It might be helpful if the author showed the code examples alongside the equivalent code in bash, so it's easy to see both what the example code is doing and how much effort is saved by doing it in dgsh.
It doesn't look all that different to me. Seems like it's just saving you mess around with assigning function inputs and outputs to shell variables. Otherwise it just looks like piping stuff around between functions.
Re: Dgsh – Directed graph shell
#46I wonder if there is any perfomance benchmark of this graph shell? Especially on some complex pipelines running huge datasets?
Re: Dgsh – Directed graph shell
#47Re: Dgsh – Directed graph shell
#48I write complex shell commands every day, but when it gets longer than 2-3 rows I switch to a text editor and write it in Perl instead. I see no need to use bash up to that complexity, doesn't look good in terminal. Poorman version of multiple pipes is to write intermediate results into files, then "cat" the files as many times as needed for the following processes. I use short file names "o1", "o2" standing for outp…
This is what it comes down to to me too. Using the shell to do programming seems to me like putting your job on hard mode. When I had to do a lot of data processing at my last job, I started building up tools in Ruby. If I had time, I'd hack the workflow so that the next time I needed it, I could just run the tool from the command line. Eventually I had a pluggable architecture that I could use to pull data from any…
Re: Dgsh – Directed graph shell
#49Earlier quoted context omitted.
Something I have found fun in the past: using xslt where the underlying document is not xml. In order for xslt to work (in java setting, apache libs) you do not need an underlying xml document, just something that satisfies the appropriate java interface. For example, you could wrap a filesystem directory structure.
Is it possible to show what XSLT is and why it's useful in like 5 minutes? I've always wanted a transformation language of some sort, but I've never managed to figure out XSLT (probably because I've never needed it) so I don't know what problems it solves or doesn't solve.
I've always wanted a transformation language of some
sort, but I've never managed to figure out XSLT
(probably because I've never needed it) so I don't
know what problems it solves or doesn't solve.
You'd be familiar with SQL. SQL is a declarative language for interacting with a relational structure. You say what you want and from where. It outputs to a table. Or, with significant effort, to more complex forms.XPath + XSLT are declarative languages for interacting with a tree structure. You say what you want, and how you want the results laid out. It's particularly useful in integration scenarios. (I need to take data in horror format X, and then transform it into the completely different horror format Y)
Example A: you have a directory full of XML files that represent streets, estates, houses, buildings and apartments across the nation. The depth of data in these nodes is inconsistent: houses are generally top-level; apartments are nested within buildings within estates. You need to (1) select one or two bedroom homes their are in a particular set of postcodes. And (2) to capture some facts about each of those homes in a completely different XML format.
XPath is useful for finding the things. XSLT is your tool for manipulating the results from the XPath query into the output document format.
Example B: a vendor sends you accounting data. They are set up to send you one nasty format only. You have a third-party internal finance system that requires a separate specific format.
Source example:
Destination example:
USD 500
USD 600
NZD 700
You could definitely knock something up that did this transform in python or perl. Particularly if you were confident where the newlines would be. There are situations where this makes sense: XML tooling is not as strong in those platforms as Java/C#, and you may want colleagues to be able to maintain this stuff without them having to learn entirely new technology stacks.However, once you're dealing with a complex problem, XSLT+XPath are what you want. If you wrote perl or python to do this, your perl or python would evolve to 80% of a slow, ill-conceived, badly implemented ripoff of the apache XPath+XSLT. And you'd run into all kinds of problems with edge-case stuff like unicode.
If I was building an editorial pipeline for a newspaper or publisher, it'd be XML+XSLT all the way. But there's a lot of places where I would avoid XML and not need XSLT.
XML is flawed for the domain where it gets the most action: system APIs. XML encourages complex, monolithic, document-separated interfaces. To correct for this, the community has layered yet more complex schema systems on top of it.
System interfaces should steer towards being tight, flat, specific, discoverable and stream-oriented. System interfaces with those qualities are easier to build and maintain and learn.
In place of XML, I prefer the approach below. At the start of your feed, assert the interface you think the other person should be receiving on. Then send messages over those vectors.
# i lines assert the interface (emphasis: this is an assertion, /not/ an IDL)
i ccy h
i account h name
i trans h date account_h ccy_h amount
i leg trans_h amount
#
# now send your data stream over those vectors
ccy USD
ccy NZD
account account/1234 "John Smith"
trans trans/0 20150808 account/1234 USD 500
trans trans/1 20150810 account/1234 USD 600
trans trans/2 20150810 account/1234 NZD 700
leg trans/2 200
leg trans/2 500
If the receiver disagrees with the interface, then it errors at startup and not half way through the stream.In this format, tree structures are possible. But you have to work for them. This nudges interfaces towards flat forms that are more greppable and awkable.
Imagine a complex business where all the interchange formats were captured in this interface script. A studious non-developer could quickly learn to really dance with it and think in terms of their data flows. They could discover things, and respond to emergencies with a text editor. You could give trusted users access to a kind of power that is rarely shared with non-developers. Users who have worked on systems like this talk of them in hushed tones that acknowledge the respect and power that was shown to them.
With XML it's harder to make reliable inferences about the schema, and harder to debug entry errors. For this reason you generally can't trust end-users with it.
Why have I gone through this? Because: if you're careful about designing your serialisation mechanisms, you can get further along before you need to resort to XSLT.
There are python3 parsing and producer mechanisms for interface script at github.com/cratuki/solent in package solent.util.interface_script (or: pip3 install solent). It wouldn't be much work to write Java/C# SAX interfaces to it.
Re: Dgsh – Directed graph shell
#50I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…
Thanks for sharing your experience. I work with Pachyderm, which is an open source data pipelining and data versioning framework. Some things like might be relevant to this conversation are the fact that Pachyderm is language agnostic and that it keeps analyses in sync with data (because it triggers off of commits to data versioning). This makes it distinct from Airflow or Luigi, for example.
Only I hope to get time to test it out in some more depth sooner rather than later (it is one of my top goals for 2017).
Also, the pipeline feature in Pachyderm does not suffer from the "dependencies between tasks rather than data" problem that I mentioned in another post here, but properly identifies separate inputs and outputs declaratively.
Pachyderm specifies workflows in a kind of DSL AFAIK, and I'm very much interested to see if it could natively fit the bill for our complex workflows. But if not, I think we can always use it in a a light-weight way to fire off scipipe workflows (instead of the applications directly), and so let scipipe take care of the complex data wiring.
We would still like to benefit from the seemingly groundbreaking "git for big data" paradigm, and auto-executed workflow on updated data, which should enable something as impactful as on-line data analyses (auto-updated upon new data) in a manageable way.