python - Adding Multiple ZODB Databases Together -


i have 3 python object databases i've constructed through zodb module, merge one. reason have 3 , not 1 because each object belongs 1 of 3 populations, , added database once code conducted analysis of said object. analysis of each object can done in parallel. code takes few days run, prevent being week long endeavor, have 3 computers each processing objects 1 of 3 populations, , outputting single zodb database once has completed. couldn't have 3 computers adding analysis of objects different populations same database because of way zodb handles conflicts. essentially, until close database, locked inside.

my questions are: 1) how can merge multiple .fs database files single master database? structure of each database same - meaning dictionary structures same between each. example, mydb may represent zodb database structure of first population:

root.['mydb']['id123456']['property1']         ...  ['id123456']['property2']         ...       ...        ...  root.['mydb']['id123457']['property1']         ...  ['id123457']['property2']         ...       ...        ...  ...  

where ellipsis represents more of same. names of keys 'property1', 'property2', etc., same each 'idxxxxxx' key within database, though values certaily vary.

2) have been smarter thing run code in parallel while still resulting in single zodb structure?

please let me know if clarification needed.

thanks!

the smarter thing have been use zeo share zodb storage among processes.

zeo shares zodb database across network , extends conflict resolution across multiple clients, can reside on same machine or elsewhere.

alternatively, use relstorage backend store zodb instead of using standard filestorage; backend uses traditional relational database provide concurrent access instead.

see zc.lockfile.lockerror in zodb usage examples either option.

the zodb data structures otherwise merely persisted python data structures; merging 3 zodb datastructures requires open each of databases , merging nested structures needed.


Comments

Popular posts from this blog

Detect support for Shoutcast ICY MP3 without navigator.userAgent in Firefox? -

web - SVG not rendering properly in Firefox -

java - JavaFX 2 slider labelFormatter not being used -