Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bug #915: Please help

nedbatchelder.com

Post reply on HN
Print view View original

1–10 of 97 posts

Re: Bug #915: Please help

#2

Post by mmastrac »

There was a similar bug in unraid that they ended up fixing in 6.8.0 where it turned out that SQLite wasn't handling some sort of error condition in read-ahead I/O? I wonder if this is related.

https://forums.unraid.net/bug-reports/prereleases/sqlite-dat...

From that report:

====== 8> In the Linux block layer each READ or WRITE can have various modifier bits set. In the case of a read-ahead you get READ|REQ_RAHEAD which tells I/O driver this is a read-ahead. In this case, if there are insufficient resources at the time this request is received, the driver is permitted to terminate the operation with BLK_STS_IOERR status. Here is an example in Linux md/raid5 driver.

> In case of Unraid it can definitely happen under heavy load that a read-ahead comes along and there are no 'stripe buffers' immediately available. In this case, instead of making calling process wait, it terminated the I/O. This has worked this way for years.

[...]

> What I suspect is that this is a bug in SQLite - I think SQLite is using direct-I/O (bypassing page cache) and issuing it's own read-aheads and their logic to handle failing read-ahead is broken. But I did not follow that rabbit hole - too many other problems to work on :/

====== 8Some related bugs they brought up:

https://bugzilla.kernel.org/show_bug.cgi?id=201685

https://patchwork.kernel.org/patch/10712695/

Edit: it appears that there's _something_ causing corruption when drivers fail read-ahead I/O. Whether it's SQLite or something else in Linux is another question.

Re: Bug #915: Please help

#3

Post by TAForObvReasons »

OT: After the lamentation of the "Github Monoculture", why was development of coveragepy moved from bitbucket to github?

https://bitbucket.org/ned/coveragepy/src/default/ original project https://nedbatchelder.com/blog/201405/github_monoculture.htm... original post https://news.ycombinator.com/item?id=7690897 HN comments

Re: Bug #915: Please help

#5

Post by x1798DE »

This is an editorialized title and as far as I know it's not even accurate. The author maintains, but did not create coverage.py.

(Edit: Original title has been changed, so this comment is no longer necessary.)

Re: Bug #915: Please help

#6

Post by geofft »

mmastrac wrote: post #2

There was a similar bug in unraid that they ended up fixing in 6.8.0 where it turned out that SQLite wasn't handling some sort of error condition in read-ahead I/O? I wonder if this is related. https://forums.unraid.net/bug-reports/prereleases/sqlite-dat... From that report: ====== 8 > In the Linux block layer each READ or WRITE can have various modifier bits set. In the case of a read-ahead you get READ|REQ_RAHEAD w…

> I think SQLite is using direct-I/O (bypassing page cache)

Should be pretty easy to confirm/deny that suspicion and see if it's worth going down the rabbit hole, no?

    strace -e open -o >(grep O_DIRECT) ./myprogram

Re: Bug #915: Please help

#7

Post by pdw »

mmastrac wrote: post #2

There was a similar bug in unraid that they ended up fixing in 6.8.0 where it turned out that SQLite wasn't handling some sort of error condition in read-ahead I/O? I wonder if this is related. https://forums.unraid.net/bug-reports/prereleases/sqlite-dat... From that report: ====== 8 > In the Linux block layer each READ or WRITE can have various modifier bits set. In the case of a read-ahead you get READ|REQ_RAHEAD w…

I don't think sqlite uses direct IO -- at least there's no mention of O_DIRECT in the source code

Re: Bug #915: Please help

#8

Post by lelf »

geofft wrote: post #6
mmastrac wrote: post #2

There was a similar bug in unraid that they ended up fixing in 6.8.0 where it turned out that SQLite wasn't handling some sort of error condition in read-ahead I/O? I wonder if this is related. https://forums.unraid.net/bug-reports/prereleases/sqlite-dat... From that report: ====== 8 > In the Linux block layer each READ or WRITE can have various modifier bits set. In the case of a read-ahead you get READ|REQ_RAHEAD w…

> I think SQLite is using direct-I/O (bypassing page cache) Should be pretty easy to confirm/deny that suspicion and see if it's worth going down the rabbit hole, no? strace -e open -o >(grep O_DIRECT) ./myprogram

  -o >(grep O_DIRECT)
This is cool. Never seen it (but it makes sense.)

Re: Bug #915: Please help

#9

Post by mmastrac »

lelf wrote: post #8
geofft wrote: post #6

Earlier quoted context omitted.

> I think SQLite is using direct-I/O (bypassing page cache) Should be pretty easy to confirm/deny that suspicion and see if it's worth going down the rabbit hole, no? strace -e open -o >(grep O_DIRECT) ./myprogram

-o >(grep O_DIRECT) This is cool. Never seen it (but it makes sense.)

Its cousin

  diff
is also very handy.

Re: Bug #915: Please help

#10

Post by supakeen »

x1798DE wrote: post #5

This is an editorialized title and as far as I know it's not even accurate. The author maintains, but did not create coverage.py. (Edit: Original title has been changed, so this comment is no longer necessary.)

I was unaware of that and it seems people have fixed the title for me. Thanks!
Post reply on HN