Big data is dead • Hacker News Board

Re: Big data is dead

#441

Post by h0p3 » Sun, Feb 12, 2023, 9:22 AM UTC

moremetadata wrote: post #137

eska wrote: post #66

Earlier quoted context omitted.

This would match what psychologists say about humans in general: we feel first, then we use our brain to justify that feeling. We’re not rational beings.

Thats because psychologists dont understand or choose to ignore how chemistry influences our personalities and emotions. An extremely simple example from the same medical/health profession is the use of SSRI's to make people feel happy. The legal system recognises how chemicals influence our feelings because of the laws that exists on illegal drugs or drink driving. The definition of rational is being informed enough…

I'd like to know more about "[Your] life under [your] state employed 'parents' and at the hands of other employees of the state"

Re: Big data is dead

#442

Post by primax » Mon, Feb 13, 2023, 6:40 AM UTC

0xB31B1B wrote: post #275

primax wrote: post #227

Earlier quoted context omitted.

I think a key driver of this is not having to use SQL. I like DynamoDB and EdgeDB because I can use a more modern and reasonable language to interact with the database.

its really difficult to do any kind of analysis without relational queries. The standard way you do this is to have an app datastore in DDB, and an ETL job that pipes your data into some data warehouse env.

EdgeDB works really well for relational queries, it's a graph native query language that renders into Postgresql. Check it out: https://www.edgedb.com/

Re: Big data is dead

#443

Post by spopejoy » Mon, Feb 13, 2023, 4:29 PM UTC

fijiaarone wrote: post #368

spopejoy wrote: post #328

Earlier quoted context omitted.

The unit test story is indeed bizarre. Done right unit tests should test the unit, and you'll never hit these problems. The villians here were monstrous test fixtures instead of mocks, "testing the fixture" instead of testing the code. Both were agency trading systems so "platforms" of a sort that needed significant refactoring to mock properly, so instead tests had to inject essentially fake concrete services. Someh…

Mocks are the personification of bad data. The only meaningful measurement derived from tests with mocks is how bad the architecture is.

I don't know what you're criticizing here. I was contrasting "mocks" and "fixtures" in the context of unit tests as ways to instrument services depended on by the code under test.

A "mock" in this paradigm is some kind of testing technology that allows you to directly instrument return values for function calls on the dependent service, whereas a "fixture" is some concrete test-only thing you coded up to use in your tests.

If a fixture just acts as a dummy return-value provider, no problem (but you probably should have used a mocking solution). The problem that arises is fixture code that simulates some or all of the production service code, and/or (even worse) allowing modification of production code to allow use as a test fixture. This is the way to madness.

Re: Big data is dead

#444

Post by inadequatespace » Sun, Feb 19, 2023, 4:18 AM UTC

diognesofsinope wrote: post #235

eska wrote: post #66

Earlier quoted context omitted.

This would match what psychologists say about humans in general: we feel first, then we use our brain to justify that feeling. We’re not rational beings.

I think the answer is simpler: people care about their careers and their family first. Think, "If the data says something that gets in the way of my career well I don't care about the data." Had the same problem when I was an economics researcher -- publication bias for what stakeholders want to hear (often the government) is rampant because that's where funding for the economics department mostly comes from.

It's only rational. The company certainly doesn't care about that individual first, as evidenced by e.g. its decision to lay them off when it doesn't think the individual is serving them, so why should the individual put the company first?

This is also known as The Iron Law of Institutions.