If you really want to, there's also the `join` command, which has been part of the standard unix toolset for longer than many of us have been alive.
Joining CSV Data Without SQL: An IP Geolocation Use Case
11–20 of 41 posts
Re: Joining CSV Data Without SQL: An IP Geolocation Use Case
#12Please don’t do this for actual work you share with other people. There’s a good reason why pandas exists.
People that do proper ETL don't use python
Re: Joining CSV Data Without SQL: An IP Geolocation Use Case
#13Please don’t do this for actual work you share with other people. There’s a good reason why pandas exists.
People that do proper ETL don't use python
Real programmers use butterflies.
Re: Joining CSV Data Without SQL: An IP Geolocation Use Case
#14You can join 2 CSV files in a GUI ETL tool like Easy Data Transform in 3 clicks:
-drag the 2 CSV files onto the canvas
-click 'Join'
-select the 2 columns to join
Re: Joining CSV Data Without SQL: An IP Geolocation Use Case
#15Ehhhhhhhh. It hasn't really made a super convincing argument not to use SQL. A lot of what is described isn't intinsicly a SQL problem, but instead an ETL problem. Eg, the use of complex types can be done by loading a csv into a TEXT table and then test typecasting on a sample set, followed by the full set. And geo indexes are no joke. Using them has made 8hr long SQL queries take seconds.
[deleted]
Re: Joining CSV Data Without SQL: An IP Geolocation Use Case
#16I‘d rather just use DuckDB and write a few SQLs, and be done.
This can be done in the terminal as well.
Re: Joining CSV Data Without SQL: An IP Geolocation Use Case
#17> a performant join that avoids the hassle of SQL tables.
Hm, a hassle is not always the same hassle for everyone. Personally, I like SQL and I enjoy the power it can leverage on all kind of data. As others wrote, working with sqlite for local data processing is a tool I do not want to miss, besides all of the great Unix coreutils and its (mostly GNU) ecosystem.
Re: Joining CSV Data Without SQL: An IP Geolocation Use Case
#18yes! non standard data wrangling, even if just for fun, is great way to gain a better standing of your workload and hardware.
tldr; [de]serialization is your bottleneck, after that it’s general data processing. both are wasting insane levels of cpu cycles. network and disk, when accessed linearly, are free.
i remember first looking into this when ec2 i3 came out, only more so since. lambda for burst cpu capacity when you can’t wait 30s for ec2 spot is interesting too.
https://nathants.com/posts/performant-batch-processing-with-...
Re: Joining CSV Data Without SQL: An IP Geolocation Use Case
#19Or, you can also use openrefine from Google.
Currently mangling a 4 GB file and working with api's that use existing data columns to provide output.. Its a great tool.
Re: Joining CSV Data Without SQL: An IP Geolocation Use Case
#20Or, you can also use openrefine from Google. Currently mangling a 4 GB file and working with api's that use existing data columns to provide output.. Its a great tool.
How does it work for much larger files?