Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"We ran out of columns"

jimmyhmiller.github.io

Post reply on HN
Print view View original

241–250 of 588 posts

Re: "We ran out of columns"

#241

Post by jimbokun »

gumby wrote: post #211
fouronnes3 wrote: post #130

Earlier quoted context omitted.

This is amazing. I can so well imagine a bright young hire joining that team, helpfully offering to "setup this thing called git" only to be laughed out of the meeting by all the "senior" staff.

Astonishingly, It took a long time for revision control to become widespread. Around 1991 when Cygnus had 6-7 employees and was based in the apartment complex where I lived, none of the GNU codebase was hosted in any sort of revision control. Everything was FTPed around as obscurely named tarballs. We had gathered something like 27 different forks of gdb floating around the net, for example. This was back when forkin…

> Pixley agreed to all of that and then ignored all of it completely.

Hahaha that's brilliant, and an important lesson for junior developers. Sometimes this is the best way forward. High risk of course. But the reward can be great.

Re: "We ran out of columns"

#242

Post by sghiassy »

hleszek wrote: post #126

When I started at my first company, they had a very complex VB application running on dozens of customers around the country, each having some particular needs of course. There was a LOT of global variables (seemingly random 4 uppercase letters) controlling everything. At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was ob…

This is nightmare-fuel. How does this happen?

As a glib answer, can I suggest, that without proper training, there were a lot of developers who had never trained under anyone or any company with proper practices??

Honest question. How does our profession root out intrinsically obvious bad practices?

Re: "We ran out of columns"

#243

Post by xpressvideoz »

Elasticsearch and OpenSearch have a similar issue. They have a soft limit on the number of "searchable" fields (called "mapping") a table can have (by the way, a table is called an "index" in their terminology. How confusing!), which is 1000. We ran into this problem because we tried to jam all the logs in every single microservice into a single table. Each microservice had a different log format, so after combining them all the number of different fields surged. We did it this way because the operations team wanted to maintain a single table, by the name of simplicity, which to this day is a reason I can't completely fathom. We were surprised because we thought Elasticsearch and OpenSearch were some kind of magic box that can somehow ingest all the data we put yet still are perfectly performant. After that incident, we introduced a common log format that applies to every microservice.

Re: "We ran out of columns"

#244

Post by cyberax »

kubectl_h wrote: post #226
cyberax wrote: post #200

Earlier quoted context omitted.

In my current company, we're using a similar approach: just shove everything into a JSON blob. If you need a constraint or an index, you can create a computed column (in PostgreSQL) that pulls out a field from JSON. For the data schema, we're using Protobufs with buf validate. This works surprisingly well, you can use the same types in the backend API and on the frontend. We even have a cron job that reads all the da…

Do you go to the trouble of updating individual values in objects using some kind of deep/partial updating function or do you just accept race conditions that come with updating full objects?

We use optimistic versioning, with a dedicated "version" field (that is actually always pulled out of the blob in all tables).

Classic fine-grained schemas are not that much different. A lot of high-level ORM frameworks simply save all the objects' fields on update, without doing fine-grained diffs.

In addition, our frontend apps also support offline mode. They can get all the relevant objects, and then operate on them locally. So our API was designed from the start to deal with conflicts.

Re: "We ran out of columns"

#245

Post by jacobyoder »

deathanatos wrote: post #154

Two early databases I worked on. The first contained monetary values. These were split over two columns, a decimal column holding the magnitude of the value, and a string column, containing an ISO currency code. Sounds good so far, right? Well, I learned much later (after, of course, having relied on the data) that the currency code column had only been added after expanding into Europe … but not before expanding int…

> But when the column was added, they just defaulted it all to USD. So and USD value could be CAD — you "just" needed to parse the address column to find out.

I bet no one even considered the idea that someone in Canada might pay in USD dollars.

I had to... discuss... with someone the possibility that if we were recording money received from a bank, we might do well to record the currency type as well, because... banks can deal with multiple currencies. "No, banks don't do that. That's not possible". This US company was getting ready to expand their services over to Europe, and I couldn't understand why no one could understand that this might be a necessary item to record.

Someone using this system in France to help finance a project in New York, for example, might want to know whether Euros or USD were the subject of the project. This was sort of a PM tool to note the cost of a project and how much was left in a budget - it wasn't moving money directly. We had a system to choose specific banks and note that 'person X moved Y currency units to the project' but... no ability to select the currency units. If it was from a UK bank, it would be GBP. A US bank, USD, etc. I was voted down as not understanding how banks worked.

Months later, a demo was presented right before launch to the people who were financing the project, and they asked how someone would know the currency being used for each project. Back came a work ticket "high priority" because we were already late for launch and multiple screens now had to accommodate this "new" requirement.

I know at least some of this is how I present info, but... being 'nice', being 'inquisitive', raising issues up the chain of command, etc.. rarely work. These sorts of extremely obvious things get ignored until the actual end users bring up the same concern. Somehow I wasn't supposed to know this sort of info because "you're a developer, you're not some international banker". Insane...

Re: "We ran out of columns"

#246

Post by NetOpWibby »

asveikau wrote: post #214
gumby wrote: post #211

Earlier quoted context omitted.

Astonishingly, It took a long time for revision control to become widespread. Around 1991 when Cygnus had 6-7 employees and was based in the apartment complex where I lived, none of the GNU codebase was hosted in any sort of revision control. Everything was FTPed around as obscurely named tarballs. We had gathered something like 27 different forks of gdb floating around the net, for example. This was back when forkin…

Linux was managed with tarballs and patch emails until a little over 20 years ago when they started using bitkeeper, then Linus wrote git.

What a legend

Re: "We ran out of columns"

#247

Post by djbusby »

cyberax wrote: post #184
djbusby wrote: post #138

Earlier quoted context omitted.

I was one of those once. Tried to get CVS in a project. Then some other dev committed 9MB of tabs 0x09 at the end of a file. Then the site was "slow" (cause the homepage was 10MB). And the blame went to...CVS somehow. I left.

> And the blame went to...CVS somehow. CVS was notorious for doing "text/binary" conversions (CR/LF line endings to CR and vice versa), sometimes inappropriately. More than once, it resulted in files where every other line was empty. I can very well see this happening several times, resulting in exponential growth of whitespaces.

Yep, that bit me/my teams more than once. But this case we had the commit from CVS showing it was just a fat-finger.

But the seniors on the team pointed to CVS. Not the individual, not the "test" process, not the deploy checks.

Nobody noticed it slow cause everything ran on LAN. Only noticed the issue on 56k

Re: "We ran out of columns"

#248

Post by lazide »

sghiassy wrote: post #242
hleszek wrote: post #126

When I started at my first company, they had a very complex VB application running on dozens of customers around the country, each having some particular needs of course. There was a LOT of global variables (seemingly random 4 uppercase letters) controlling everything. At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was ob…

This is nightmare-fuel. How does this happen? As a glib answer, can I suggest, that without proper training, there were a lot of developers who had never trained under anyone or any company with proper practices?? Honest question. How does our profession root out intrinsically obvious bad practices?

It happens because it’s easier - until it’s impossible, anyway.

The training and best practices you’re talking about is learned experience about how to avoid it getting impossible. But that almost always involves expense that the business side considers ‘stupid’.

Re: "We ran out of columns"

#249

Post by rblatz »

llmblockchain wrote: post #160
fouronnes3 wrote: post #130

Earlier quoted context omitted.

This is amazing. I can so well imagine a bright young hire joining that team, helpfully offering to "setup this thing called git" only to be laughed out of the meeting by all the "senior" staff.

I've been that person a few times. 1. The only developer on the team with Github and put forward the idea of the company not hosting their own source code with TFS. 2. The only developer using branches with git when the co-founder asked (demanded) everyone to only use master. The list goes on!

There are advantages to trunk based development, not exactly only using master, but close enough that you could probably spin it to comply.

Re: "We ran out of columns"

#250

Post by insane_dreamer »

hleszek wrote: post #126

When I started at my first company, they had a very complex VB application running on dozens of customers around the country, each having some particular needs of course. There was a LOT of global variables (seemingly random 4 uppercase letters) controlling everything. At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was ob…

How long ago was this?
Post reply on HN