I know that you wrote that you used flat files for Viaweb. I was wondering if you were still doing that for YC News, if you had broken down and started using a database or rolled your own data store with Arc.
Ask PG: Database, flat files or other for YC News?
1–10 of 15 posts
Re: Ask PG: Database, flat files or other for YC News?
#2This has been covered before. IIRC, it's all in memory. The data structures are saved to disk as s-expressions and read on application startup.
Re: Ask PG: Database, flat files or other for YC News?
#3This has been covered before. IIRC, it's all in memory. The data structures are saved to disk as s-expressions and read on application startup.
[deleted]
Re: Ask PG: Database, flat files or other for YC News?
#4This has been covered before. IIRC, it's all in memory. The data structures are saved to disk as s-expressions and read on application startup.
Right. Except now to make restarts faster I use lazy loading.
Re: Ask PG: Database, flat files or other for YC News?
#5This has been covered before. IIRC, it's all in memory. The data structures are saved to disk as s-expressions and read on application startup.
Right. Except now to make restarts faster I use lazy loading.
Please write about how you do this and how you interact with the files (loading and serializing). I'm looking at alternatives to using relational databases and information about how to avoid data corruption (features analogous to transactions) is scare. How would you convince a mission critical site developer that this is safe?
Re: Ask PG: Database, flat files or other for YC News?
#6Earlier quoted context omitted.
Right. Except now to make restarts faster I use lazy loading.
Please write about how you do this and how you interact with the files (loading and serializing). I'm looking at alternatives to using relational databases and information about how to avoid data corruption (features analogous to transactions) is scare. How would you convince a mission critical site developer that this is safe?
Transaction isolation would be handled like any non-database backed application: use your language and/or library's native thread synchronization features.
As for persisting transactions, you could marshall to file the deltas for each transaction, and on regular intervals apply them to the full image to create an up-to-date image.
Re: Ask PG: Database, flat files or other for YC News?
#7Earlier quoted context omitted.
Please write about how you do this and how you interact with the files (loading and serializing). I'm looking at alternatives to using relational databases and information about how to avoid data corruption (features analogous to transactions) is scare. How would you convince a mission critical site developer that this is safe?
Transaction isolation would be handled like any non-database backed application: use your language and/or library's native thread synchronization features. As for persisting transactions, you could marshall to file the deltas for each transaction, and on regular intervals apply them to the full image to create an up-to-date image.
> use your language and/or library's native thread synchronization features
You may try using a single thread and cooperative multitasking. It helps if your language makes this convenient, e.g., Stackless Python or Scheme:
http://news.ycombinator.com/item?id=45561
Just remember to yield every now and then if you do anything lengthy. Depending on your application, it may not be that bad, and if you don't need anything fancier in the way of scheduling fairness, it makes your life really simple.
Re: Ask PG: Database, flat files or other for YC News?
#8Earlier quoted context omitted.
Right. Except now to make restarts faster I use lazy loading.
Please write about how you do this and how you interact with the files (loading and serializing). I'm looking at alternatives to using relational databases and information about how to avoid data corruption (features analogous to transactions) is scare. How would you convince a mission critical site developer that this is safe?
No idea how pg handles it, but here is an easy do-it-yourself way:
http://armstrongonsoftware.blogspot.com/2006/09/pure-and-sim...
It doesn't matter it is in Erlang - you can do it in any language. A Lisp that implements software transactional memory is Clojure (runs on the JVM): http://clojure.sourceforge.net/
Re: Ask PG: Database, flat files or other for YC News?
#9Earlier quoted context omitted.
Transaction isolation would be handled like any non-database backed application: use your language and/or library's native thread synchronization features. As for persisting transactions, you could marshall to file the deltas for each transaction, and on regular intervals apply them to the full image to create an up-to-date image.
> use your language and/or library's native thread synchronization features You may try using a single thread and cooperative multitasking. It helps if your language makes this convenient, e.g., Stackless Python or Scheme: http://news.ycombinator.com/item?id=45561 Just remember to yield every now and then if you do anything lengthy. Depending on your application, it may not be that bad, and if you don't need anything…
Yep, the CSP style of channel communication is great. Those thinking Erlang is better than sliced bread need to make sure they're up on what came before and after; http://swtch.com/~rsc/thread/ Personally, I find Erlang to be too clunky as a language. Good for special purpose telephone switching software maybe, but for general programming the CSP style can be done in nicer ways than having to switch to a whole new language.
Re: Ask PG: Database, flat files or other for YC News?
#10Earlier quoted context omitted.
> use your language and/or library's native thread synchronization features You may try using a single thread and cooperative multitasking. It helps if your language makes this convenient, e.g., Stackless Python or Scheme: http://news.ycombinator.com/item?id=45561 Just remember to yield every now and then if you do anything lengthy. Depending on your application, it may not be that bad, and if you don't need anything…
Yep, the CSP style of channel communication is great. Those thinking Erlang is better than sliced bread need to make sure they're up on what came before and after; http://swtch.com/~rsc/thread/ Personally, I find Erlang to be too clunky as a language. Good for special purpose telephone switching software maybe, but for general programming the CSP style can be done in nicer ways than having to switch to a whole new la…
Not often we hear about CSP here. I did my doctorate in it (Hoare was the head of department at the time). Are you using CSP for something real?