bigdata - 1 GB Data with key and value, what kind of data structure to store them? 1TB? 1 PB? -

-

there 1 gb data key , value, kind of data structure store them? if data increase 1tb? 1 pb? need access them every day. , how long need access them? true time? 1 minute? 1 hour?

i answered using hashtable in 1gb , 1tb. few seconds take? i'm not sure how calculate true time. when coming 1 pb, can sort data , divide them several part , store them in hashtables.

seems interviewer not satisfied answer. seems i'm totally wrong :(

imho, choice of structure depends heavily on how memory have. ram totally out of question when 1tb or 1pb. when interviewers ask questions these, trying see how @ logical thinking rather expecting spot-on, exact solution(at least feel this).

coming actual question, use distributed platform, hadoop, sreejith has said. in systems hadoop use multiple systems single system in order leverage combined power gain better performance. approach can reduce read/write time compared single machine, if powerful ram , processor. along hadoop provides data structures sequencefile makes easy store , process huge datasets.

but whatever method choose, disk based access(which cannot avoided when dealing data of ~tb or ~pb) slower memory based access. so, need choose data structure allows minimize disk accesses as possible. see paper detailed info on i'm trying say.

hth


Comments

Post a Comment

Popular posts from this blog

java - JavaFX 2 slider labelFormatter not being used -

-
i have following code not label slider "foo50", etc... 1 expect setlabelformatter call, instead seems have no effect (the slider labelled 50, 100, etc , println's dont run). there missing? not implemented in default 2.2 skin? using oracle java 7 package javafxbugtest; import javafx.application.application; import javafx.scene.*; import javafx.scene.control.*; import javafx.stage.*; import javafx.util.stringconverter; public class javafxbugtest extends application { @override public void start(final stage primarystage) { slider cp = new slider(); cp.setlabelformatter(new stringconverter<double>() { @override public string tostring(double t) { system.out.println("nope?"); return "foo" + t.tostring(); } @override public double fromstring(string string) { system.out.println("he
Read more

Detect support for Shoutcast ICY MP3 without navigator.userAgent in Firefox? -

-
the current version of mozilla firefox 23.0.1, version not support play mp3 shoutcast streams tcp port different 80 (most common 8000 shoutcast 1.9.8). i use flash when mp3 support not available in html5 audio, way detect is: try{ var = document.createelement('audio'); r = !!(a.canplaytype && !!a.canplaytype("audio/mpeg; codecs=mp3").replace(/^no$/,'')) }catch(e){ r = false; } the support mp3 shoutcast streams in firefox added in version 24 . a.canplaytype("audio/mpeg; codecs=mp3") = in chrome , firefox, chrome support, firefox not support, due current code detect not work firefox. the current version of jquery support ie 6 1.10.2, version not has .browser i think "stylized" way testing features , not querying browsers / versions, notwithstanding here see hard not violate "principle". what "stylized" way of detect mp3 icy support without navigator.useragent in firefox? ther
Read more

web - SVG not rendering properly in Firefox -

-
i have problem svg renderring in firefox. whilst works in safari, chrome or in ie, doesn't work in firefox. inspecting element seems layers there, background visible if it's z-index higher z-index of others. rendered in firefox: svg logo in firefox link original source file: svg file thanks help. ps: used sketch 3 export svg. if helps somehow. you're experiencing bug 995813 . i fixed bug time ago change still making way through release process example display correctly firefox 31 onwards released on 22 july 2014. in meantime if move fill="white" <mask> element child <use> element work around firefox bug.
Read more