Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Browserball

weareinstrument.com

Post reply on HN
Print view View original

41–50 of 77 posts

Re: Browserball

#41

Post by DonHopkins »

agumonkey wrote: post #38
DonHopkins wrote: post #35

Earlier quoted context omitted.

Then you could make the ball roll down towards the corner and drop into another window, of course! The NeWS window system let you reshape the entire framebuffer to any orientation or clipping, as well as individual windows and sub-windows, in the late 1980's. Why shouldn't I be able to lay down on my side next to a laptop and read a web page off the screen sideways, or adjust the rotation of the window to match the i…

Why did I not read your nickname .. :) The pseudo billboard idea is cute, but I wonder how it fares in readability on non HD screen (font rendering in 3D space may impede it, then viewing angles.). I'd love to see it done anyway, it's cute idea. Another 'this is ' rant, we should have way more sensors on usb or i2c and have our laptops integrate with the real world more.

Here are some ideas and discussion about aQuery -- like jQuery for accessibility [1] -- which would be useful for implementing this kind of stuff out of of existing window systems and desktop applications.

[1] aQuery: http://donhopkins.com/mediawiki/index.php/AQuery

Morgan Dixon's work is truly breathtaking and eye opening, and I would love for that to be a core part of a scriptable hybrid Screen Scraping / Accessibility API approach.

Screen scraping techniques are very powerful, but have limitations. Accessibility APIs are very powerful, but have different limitations. But using both approaches together, screencasting and re-composing visual elements, and tightly integrating it with JavaScript, enables a much wider and interesting range of possibilities.

Think of it like augmented reality for virtualizing desktop user interfaces. The beauty of Morgan's Prefab is how it works across different platforms and web browsers, over virtual desktops, and how it can control, sample, measure, modify, augment and recompose guis of existing unmodified applications, even dynamic language translation, so they're much more accessible and easier to use!

----

James Landay replies:

Don,

This is right up the alley of UW CSE grad student Morgan Dixon. You might want to also look at his papers.

Don emails Morgan Dixon:

Morgan, your work is brilliant, and it really impresses me how far you've gone with it, how well it works, and how many things you can do with it!

I checked out your web site and videos, and they provoked a lot of thought so I have lots of questions and comments.

I really like the UI Customization stuff, and also the sideviews!

Combining your work with everything you can do with native accessibility APIs, in an HTML/JavaScript based, user-customizable, scriptable, cross platform user interface builder like (but transcending) HyperCard would be awesome!

I would like to discuss how we could integrate Prefab with a Javascriptable, extensible API like aQuery, so you could write "selectors" that used prefab's pattern recognition techniques, bind those to JavaScript event handlers, and write high level widgets on top of that in JavaScript, and implement the graphical overlays and gui enhancements in HTML/Canvas/etc like I've done with Slate and the WebView overlay.

Users could literally drag controls out of live applications, plug them together into their own "stacks", configure and train and graphically customize them, and hook them together with other desktop apps, web apps and services!

For example, I'd like to make a direct manipulation pie menu editor, that let you just drag controls out of apps and drop them into your own pie menus, that you can inject into any application, or use in your own guis. If you dragged a slider out of an app into the slice of a pie menu, it could rotate it around to the slice direction, so that the distance you moved from the menu center controlled the slider!

While I'm at it, here's some stuff I'm writing about the jQuery Pie Menus. http://donhopkins.com/mediawiki/index.php/JQuery_Pie_Menus

----

Web Site: Morgan Dixon's Home Page. http://morgandixon.net/

Web Site: Prefab: The Pixel-Based Reverse Engineering Toolkit. https://web.archive.org/web/20130104165553/http://homes.cs.w...

Video: Prefab: What if We Could Modify Any Interface? Target aware pointing techniques, bubble cursor, sticky icons, adding advanced behaviors to existing interfaces, independent of the tools used to implement those interfaces, platform agnostic enhancements, same Prefab code works on Windows and Mac, and across remote desktops, widget state awareness, widget transition tracking, side views, parameter preview spectrums for multi-parameter space exploration, prefab implements parameter spectrum preview interfaces for both unmodified Gimp and Photoshop: http://www.youtube.com/watch?v=lju6IIteg9Q

PDF: A General-Purpose Target-Aware Pointing Enhancement Using Pixel-Level Analysis of Graphical Interfaces. Morgan Dixon, James Fogarty, and Jacob O. Wobbrock. (2012). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '12. ACM, New York, NY, 3167-3176. 23%. https://web.archive.org/web/20150714010941/http://homes.cs.w...

Video: Content and Hierarchy in Prefab: What if anybody could modify any interface? Reverse engineering guis from their pixels, addresses hierarchy and content, identifying hierarchical tree structure, recognizing text, stencil based tutorials, adaptive gui visualization, ephemeral adaptation technique for arbitrary desktop interfaces, dynamic interface language translation, UI customization, re-rendering widgets, Skype favorite widgets tab: http://www.youtube.com/watch?v=w4S5ZtnaUKE

PDF: Content and Hierarchy in Pixel-Based Methods for Reverse-Engineering Interface Structure. Morgan Dixon, Daniel Leventhal, and James Fogarty. (2011). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '11. ACM, New York, NY, 969-978. 26%. https://web.archive.org/web/20150714010931/http://homes.cs.w...

Video: Sliding Widgets, States, and Styles in Prefab. Adapting desktop interfaces for touch screen use, with sliding widgets, slow fine tuned pointing with magnification, simulating rollover to reveal tooltips: https://www.youtube.com/watch?v=8LMSYI4i7wk

Video: A General-Purpose Bubble Cursor. A general purpose target aware pointing enhancement, target editor: http://www.youtube.com/watch?v=46EopD_2K_4

PDF: Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. Morgan Dixon and James Fogarty. (2010). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '10. ACM, New York, NY, 1525-1534. 22% https://web.archive.org/web/20150714010936/http://homes.cs.w...

PDF: Prefab: What if Every GUI Were Open-Source? Morgan Dixon and James Fogarty. (2010). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '10. ACM, New York, NY, 851-854. https://web.archive.org/web/20141024012013/http://homes.cs.w...

Morgan Dixon's Research Statement: http://morgandixon.net/morgan-dixon-research-statement.pdf

Community-Driven Interface Tools

Today, most interfaces are designed by teams of people who are collocated and highly skilled. Moreover, any changes to an interface are implemented by the original developers and designers who own the source code. In contrast, I envision a future where distributed online communities rapidly construct and improve interfaces. Similar to the Wikipedia editing process, I hope to explore new interface design tools that fully democratize the design of interfaces. Wikipedia provides static content, and so people can collectively author articles using a very basic Wiki editor. However, community-driven interface tools will require a combination of sophisticated programming-by-demonstration techniques, crowdsourcing and social systems, interaction design, software engineering strategies, and interactive machine learning.

Re: Browserball

#42

Post by sleepychu »

Cpoll wrote: post #21
sleepychu wrote: post #9

Earlier quoted context omitted.

How do you know the coordinates are absolute? It seems like the child windows are 'special', perhaps the web page can obtain the relative coordinates of these child panels?

It's using window.screenY (and screenX, of course,) which gives absolute coordinates. https://developer.mozilla.org/en-US/docs/Web/API/Window/scre... I don't think the method you're describing exists. If you want child coordinates relative to parent coordinates, you would use both the parent and child's absolute coordinates.

Thanks to you (and others who've provided this information), so I guess my question now is what is the privacy leak described. I understand that the browser is confessing information about the machine to the web page but I don't understand why that represents a private piece of information. I feel like the information about where the window is located on the screen is useful for implementation and I can't think what it says about me as a user.

Re: Browserball

#43

Post by tambourine_man »

This is amazing.

Kind of like when I first dragged a window across multiple monitors in the 90s. We are so used to content being stuck in artificial containers that it's kind of crazy when it transcends them.

Re: Browserball

#44

Post by beeboop »

anc84 wrote: post #36
ErikRogneby wrote: post #25

Earlier quoted context omitted.

Might as well say don't use the web. Disabling javascript will break most sites with a few rare exceptions. (like HN)

It's actually quite the opposite. Most sites will become lightning fast and distraction free.

I have use NoScript consistently for years and I disagree with you. Essentially the only thing NoScript does is make it so when you haven't visited a website before, it doesn't automatically trust it. For the most part every single website I go to needs to have JS enabled for anything to work beyond just reading content.

And even then, I would say 70% of the time, some critical piece of content on a website does not work with JS disabled, be it images or text or video or etc.

Re: Browserball

#45

Post by beeboop »

zbjornson wrote: post #30
cyphar wrote: post #16

I remember playing with this at least 8 years ago. It's really scary that there's enough information disclosure in browsers that it's possible to do this.

What do you see being disclosed? The origin has control over the windows it creates and nothing else.

You can very uniquely identify people based on browser positioning and sizing.

This tester uses it I believe: https://panopticlick.eff.org/

Re: Browserball

#46

Post by aidenn0 »

nxzero wrote: post #8
21 wrote: post #4

What's the reason for allowing web pages to get absolute screen coordinates? This is a privacy leak. I have a 24" screen, and I don't keep the browser window maximized because it would be too big. I presume other people do to, and I'm pretty sure most have a preferred size and position.

Right, never max out your window, it's a known data leak used to fingerprint users.

You can get screen resolution from JS, so I don't see how maxing out the window makes any difference.

Re: Browserball

#47

Post by goblinking »

TazeTSchnitzel wrote: post #26
21 wrote: post #4

What's the reason for allowing web pages to get absolute screen coordinates? This is a privacy leak. I have a 24" screen, and I don't keep the browser window maximized because it would be too big. I presume other people do to, and I'm pretty sure most have a preferred size and position.

> What's the reason for allowing web pages to get absolute screen coordinates? Back in the 90's, Microsoft and Netscape were all too happy to give JS developers the world with almost no regard for security consequences. We've spent the last 20 years trying to fix their mistakes.

are you SURE? because WebUSB and other abomination are a rising tide of diarrhea

Re: Browserball

#48

Post by detaro »

aidenn0 wrote: post #46
nxzero wrote: post #8

Earlier quoted context omitted.

Right, never max out your window, it's a known data leak used to fingerprint users.

You can get screen resolution from JS, so I don't see how maxing out the window makes any difference.

A screen resolution is less unique than the size of a maximized window.

Re: Browserball

#50

Post by danielweber »

beeboop wrote: post #44
anc84 wrote: post #36

Earlier quoted context omitted.

It's actually quite the opposite. Most sites will become lightning fast and distraction free.

I have use NoScript consistently for years and I disagree with you. Essentially the only thing NoScript does is make it so when you haven't visited a website before, it doesn't automatically trust it. For the most part every single website I go to needs to have JS enabled for anything to work beyond just reading content. And even then, I would say 70% of the time, some critical piece of content on a website does not…

And then starts the crazy hunt for the one thing you need to turn on to make the page work.

I was trying many years ago to create a public DB/wiki telling us which things we need to turn on to get the page to work, but it got abandoned before I really started.

Post reply on HN