Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automatic Generation of Visualizations and Infographics with LLMs

microsoft.github.io

Post reply on HN
Print view View original

41–50 of 56 posts

Re: Automatic Generation of Visualizations and Infographics with LLMs

#41

Post by esafak »

pizza wrote: post #35
javajosh wrote: post #24

No, absolutely not. How can you trust the output from such a black box system? Who is to say that the LLM won't add or remove data points to make the chart "look good"? Heaven help us if decision makers start taking this output seriously. But of course they will, because the charts will look professional and plausible, because that's what the prompt requires.

How do you trust matplotlib? Same way: if you need to audit plots, audit the generated source code.

So instead of auding MPL once (or never because MPL doesn't have a habit of broken output) I should audit the output of this LLM for every query because it does have a habit of hallucinating?

Re: Automatic Generation of Visualizations and Infographics with LLMs

#42

Post by dylan604 »

w-m wrote: post #5

Last week I helped someone organizing and analyzing their data in Excel. As I'm using Excel only once every couple of years, I had to rewatch the wonderful "You Suck at Excel with Joel Spolsky" to be productive again. Now seeing this announcement page, I was immediately reminded of the mini-rant towards the end of the video [0]: > On average, once every three months, there's a startup that makes a thing that they say…

But nobody today is going to read let alone promote a blog about pivot tables. Sprinkle in LLM references, and the fad wave riders will sing its praises

Re: Automatic Generation of Visualizations and Infographics with LLMs

#43

Post by nologic01 »

andybak wrote: post #39
nologic01 wrote: post #37

Earlier quoted context omitted.

Possible is not the same as admissible. Good to know that the prevailing commercial tech culture now sees plagiarism and stealing ideas without attribution as the modern way of doing business and hopes that dressing things up under some algorithmic veil will hide the act. I guess the pit of moral decline has no bottom. The consolation is that theft has never been the road to wealth. Once the plundering is over the on…

I'm afraid I'm just unclear on exactly what part of this you argue is crossing a moral line. I.e. what is being stolen without attribution? I'm genuinely not getting what you mean in this specific case.

Limited visualization grammar means that any non-trivial visualization request will be lifting a particular solution, more or less verbatim.

Re: Automatic Generation of Visualizations and Infographics with LLMs

#45

Post by ryanklee »

nologic01 wrote: post #43
andybak wrote: post #39

Earlier quoted context omitted.

I'm afraid I'm just unclear on exactly what part of this you argue is crossing a moral line. I.e. what is being stolen without attribution? I'm genuinely not getting what you mean in this specific case.

Limited visualization grammar means that any non-trivial visualization request will be lifting a particular solution, more or less verbatim.

I don't see how it's possible to show that the solution is lifted by the LLL as opposed to a arrived at by the LLM.

It seems to me that such solutions are soon to be within the set potentially constructed by an LLM.

Re: Automatic Generation of Visualizations and Infographics with LLMs

#46

Post by nologic01 »

ryanklee wrote: post #45
nologic01 wrote: post #43

Earlier quoted context omitted.

Limited visualization grammar means that any non-trivial visualization request will be lifting a particular solution, more or less verbatim.

I don't see how it's possible to show that the solution is lifted by the LLL as opposed to a arrived at by the LLM. It seems to me that such solutions are soon to be within the set potentially constructed by an LLM.

As they say, people are unwilling to understand something if their monetary gain depends on not understanding it.

Let me break it down for you. If I ask for a visualization that squares the circle and there is one repo that has an example of squaring the circle, the LLM will "arrive" at a way of squaring the circle.

Re: Automatic Generation of Visualizations and Infographics with LLMs

#47

Post by ryanklee »

nologic01 wrote: post #46
ryanklee wrote: post #45

Earlier quoted context omitted.

I don't see how it's possible to show that the solution is lifted by the LLL as opposed to a arrived at by the LLM. It seems to me that such solutions are soon to be within the set potentially constructed by an LLM.

As they say, people are unwilling to understand something if their monetary gain depends on not understanding it. Let me break it down for you. If I ask for a visualization that squares the circle and there is one repo that has an example of squaring the circle, the LLM will "arrive" at a way of squaring the circle.

That's not really answering my question.

If (1) an LLM is able to arrive at solutions in the same class of difficulty as the solution for the target problem and (2) it's not possible to establish the provenance of the solution actually offered by the LLM, then what's the argument for assuming that the solution is based on IP rather than constructive reasoning?

Re: Automatic Generation of Visualizations and Infographics with LLMs

#48

Post by bugglebeetle »

w-m wrote: post #5

Last week I helped someone organizing and analyzing their data in Excel. As I'm using Excel only once every couple of years, I had to rewatch the wonderful "You Suck at Excel with Joel Spolsky" to be productive again. Now seeing this announcement page, I was immediately reminded of the mini-rant towards the end of the video [0]: > On average, once every three months, there's a startup that makes a thing that they say…

Even Microsoft has to know Excel is shit software for large datasets. I can’t even get it to do a VLOOKUP correct half the time.

Re: Automatic Generation of Visualizations and Infographics with LLMs

#49

Post by vykthur »

computerex wrote: post #31
vimesy wrote: post #30

Earlier quoted context omitted.

"You are a helpful assistant highly skilled in writing PERFECT code for visualizations. Given some code template, you complete the template to generate a visualization given the dataset and the goal described. The code you write MUST FOLLOW VISUALIZATION BEST PRACTICES ie. meet the specified goal, apply the right transformation, use the right visualization type, use the right data encoding, and use the right aestheti…

Telling the LLM that it must do something is not a guarantee that it'll follow through.

True. This is an open area of research. Tools like guidance (or other implementations of constrained decoding with llms [1,2]) will likely help improve this problem.

[1] A guidance language for controlling large language models. https://github.com/guidance-ai/guidance

[2] Knowledge Infused Decoding https://arxiv.org/abs/2204.03084

Re: Automatic Generation of Visualizations and Infographics with LLMs

#50

Post by phillipcarter »

javajosh wrote: post #24

No, absolutely not. How can you trust the output from such a black box system? Who is to say that the LLM won't add or remove data points to make the chart "look good"? Heaven help us if decision makers start taking this output seriously. But of course they will, because the charts will look professional and plausible, because that's what the prompt requires.

> Who is to say that the LLM won't add or remove data points to make the chart "look good"?

I don't think you're thinking creatively enough here. A good system that makes use of these concept (because it's a research project, not a product!) will likely ensure that actions the LLM takes are non-destructive and inherently undoable. For example, if the underlying data was changed by the LLM, you can statically verify that and show a warning, emit an error, or ... something else entirely!

Post reply on HN