Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using the Dropbox Datastore API in Python

dropbox.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Using the Dropbox Datastore API in Python

#21

Post by krallin »

smarx wrote: post #19
krallin wrote: post #17

I wonder whether it's intentional that the tutorial isn't "copy-pastable". Specifically, it does not spell out any of the imports, and uses very large except clauses.

Hi, I wrote the tutorial. :-) In general, I based the tutorial on the sample app that ships with the SDK. That app uses Flask and presents some actual UI. If you want full working code, I would suggest taking a look at that sample. The tutorial is basically fragments from that sample that show the basic concepts, but without some boilerplate, the fragments are not themselves runnable. (BTW, where are the "very large…

Ha! Nice to see you here : )

The except clause is in the "dropbox_auth_finish" view. That might be personal paranoia, but I'm sure that every single timeI think "It's OK to put a `except:` here", it eventually comes back to bite me (without exception ;) )!

I do understand the motivation to keep it minimal (and I think the Flask boilerplate is indeed well understood), but putting together the sample without the Dropbox SDK imports (specifically without an IDE) might end up being a bit more bothersome than optimal : )

Just personal opinion, of course!

Re: Using the Dropbox Datastore API in Python

#22

Post by smarx »

krallin wrote: post #21
smarx wrote: post #19

Earlier quoted context omitted.

Hi, I wrote the tutorial. :-) In general, I based the tutorial on the sample app that ships with the SDK. That app uses Flask and presents some actual UI. If you want full working code, I would suggest taking a look at that sample. The tutorial is basically fragments from that sample that show the basic concepts, but without some boilerplate, the fragments are not themselves runnable. (BTW, where are the "very large…

Ha! Nice to see you here : ) The except clause is in the "dropbox_auth_finish" view. That might be personal paranoia, but I'm sure that every single timeI think "It's OK to put a `except:` here", it eventually comes back to bite me (without exception ;) )! I do understand the motivation to keep it minimal (and I think the Flask boilerplate is indeed well understood), but putting together the sample without the Dropbo…

Oh, I misunderstood "very large." :-) The actual code sample has a more detailed list of exceptions that it handles, but the code was a bit long.

Re: Using the Dropbox Datastore API in Python

#23

Post by tonyplee »

smarx wrote: post #18
tonyplee wrote: post #16

Earlier quoted context omitted.

Thx, smarx Good info! 10MB database limit is too small - somehow I would image dropbox wants app to create hundreds of MB if not GB of data. :-) Are there any test code available? Open Source / on github? It is much easier to understand the flow from the test code than API doc. If test code is available, it is easily to morph that info benchmark type code.

The Python library is open source, but no, I don't think we have any open source test code. Keep in mind that file data should go outside of datastores (in files), so the 10MB is for the per-user, per-app structured data. E.g. contacts, app settings, game state, etc. 10MB actually goes a pretty long way there. Also keep in mind that, at present, all of the SDKs load the entire datastore into memory, so there's a pret…

Thx again for the info.

I see why the 10MB limit.

What's the Pro/Con of this compare to just use sqlite on a dropbox volume?

From what I read, the API doesn't do any operational transformation such as merge, delta the changes' etc. The client app is expected to do it.

Re: Using the Dropbox Datastore API in Python

#24

Post by smarx »

tonyplee wrote: post #23
smarx wrote: post #18

Earlier quoted context omitted.

The Python library is open source, but no, I don't think we have any open source test code. Keep in mind that file data should go outside of datastores (in files), so the 10MB is for the per-user, per-app structured data. E.g. contacts, app settings, game state, etc. 10MB actually goes a pretty long way there. Also keep in mind that, at present, all of the SDKs load the entire datastore into memory, so there's a pret…

Thx again for the info. I see why the 10MB limit. What's the Pro/Con of this compare to just use sqlite on a dropbox volume? From what I read, the API doesn't do any operational transformation such as merge, delta the changes' etc. The client app is expected to do it.

The main pro is the automatic merging. "The API" is a fuzzy term here. There's an interaction between a client and a server, and the client is running an SDK. In the case of the Datastore API, the server doesn't perform any merging or OT, but the client SDK does. Your code just gets conflict resolution for free. The Python SDK is a bit of a special case in that it doesn't implement this logic, but the others (JavaScript, iOS, Android) do.

In contrast, if you make a change to a sqlite database on two different devices, you now have two different files and no way to merge them. (There are people who have used Dropbox to sync sqlite databases this way, and they've ended up writing diff/patch over sqlite to merge changes. Using datastores is a lot simpler and more likely to be correct.)

Re: Using the Dropbox Datastore API in Python

#25

Post by tonyplee »

smarx wrote: post #24
tonyplee wrote: post #23

Earlier quoted context omitted.

Thx again for the info. I see why the 10MB limit. What's the Pro/Con of this compare to just use sqlite on a dropbox volume? From what I read, the API doesn't do any operational transformation such as merge, delta the changes' etc. The client app is expected to do it.

The main pro is the automatic merging. "The API" is a fuzzy term here. There's an interaction between a client and a server, and the client is running an SDK. In the case of the Datastore API, the server doesn't perform any merging or OT, but the client SDK does. Your code just gets conflict resolution for free. The Python SDK is a bit of a special case in that it doesn't implement this logic, but the others (JavaScr…

Hmmmmm, my understanding of OT is that you have to implement that on Server.....

Only the server has understanding of the most current states of the dataset and can try to sync up with multiple client at the same time.

Re: Using the Dropbox Datastore API in Python

#26

Post by smarx »

tonyplee wrote: post #25
smarx wrote: post #24

Earlier quoted context omitted.

The main pro is the automatic merging. "The API" is a fuzzy term here. There's an interaction between a client and a server, and the client is running an SDK. In the case of the Datastore API, the server doesn't perform any merging or OT, but the client SDK does. Your code just gets conflict resolution for free. The Python SDK is a bit of a special case in that it doesn't implement this logic, but the others (JavaScr…

Hmmmmm, my understanding of OT is that you have to implement that on Server..... Only the server has understanding of the most current states of the dataset and can try to sync up with multiple client at the same time.

This is incorrect. OT can indeed be implemented on the client.

The way it works in the Datastore API is that the client sends its changes to the server with an attached "parent revision." If that parent revision is the revision that the server has, then the change goes through (no conflict). If the revision doesn't match, then the change is rejected by the server, and it's up to the client to pull down the latest changes from the server, merge things (via, in the case of lists, OT), and then try again.

Post reply on HN