Viewing profile — adrianbr

[flagged]

demo

Hey folks, dltHub, maker of dlt "data load tool" is releasing a LLM-native workflow for creating pipelines. This is OSS and you can try it yourself (15-30min is a good time) Is thi…

The whole idea is pretty nuts! I can imagine it being used as the dev env for teams that use SQLMesh and can thus port the sql. Might be worth investigating with them

Dear HN community, I’m Adrian, one of the creators of dlt, the most popular OSS python library for building data ingestion pipelines. It automates all the tedious data engineering …

That's really cool, did you already saw the dlt library? That one's done for very easy to use EL in python. It's similarly modular and built by senior data engineers for the data t…

congrats on the hard work and this launch!

ahh good old manual fine tuning and maintenance. We are adding data contracts for things like event ingeston where schema needs to be strict or cases where you know ahead of time w…

Thank you! That's the example we looked at for our dlt-airflow integration :) the dlt dag becomes an airflow dag.

Duckdb is analytical and gained popularity with the analytics crowd. it has multiple features that make it play well with use cases in that ecosystem such as aggregation speed, par…

for now :) Thanks for pointing it out - and it looks like we should add an aws lambda guide too :) If you want to deploy to lambda, try asking in the slack community, some folks th…

Thank you for the heads up! it is unfortunate, and with 3 letter acronyms this will happen. An easy way to remember is that we are the one you can pip install and plays well in the…

This is amazing! to figure out the website apis has always been a huge pita. With our dlt library project we can turn the openapi spec into pipelines and have the data pushed somew…

There are multiple ways to run together - we will show a few in a demo coming out soon. We also consider a tighter integration like with Airflow described here as a possible next s…

you can request a source or a feature by opening an issue on sources/dlt repo https://github.com/dlt-hub

Since dlt generates a schema, and tracks evolution etc, contains lineage, and follows data vault standard it can easily provide metdata or lineage info to the other tools. At the s…

Thank you for the feedback! I can see now how it could be confusing. The reason we used chatgpt is because it's an easy starting point - why read through examples when you can get …

dlt is a python library that you can probably plug into the OpenRefine java application to enable moving the data somewhere easily and into different formats, making OpenRefine mor…