Java Huge csv file processing and storing using Apache Spark/ Kafka/ Storm to Cassandra -


i working on requirement need read sensor data csv/tsv , insert cassandra db.

csv format:

sensor1 timestamp1 value
sensor1 timestamp2 value
sensor2 timestamp1 value
sensor2 timestamp3 value

details:

user can upload file our web application. once file uploaded, need display unique values column user in next page. for example ->

  1. sensor1 node1
  2. sensor2 node2
  3. sensorn create

user can either map sensor1 existing primary key called node1, in case timestamps , values sensor1 added table primary key equal node1 or create primary key, in case timestamps , values added new primary key.

i able implement using java8 streaming , collection. working small csv file.

question:

  1. how can upload huge csv/ tsv file (200 gb) web application? shall upload file in hdfs , specify path in ui? have split huge file small chunks (50 mb each).

  2. how can unique values first column? can use kafka/ spark here? need insert timestamp/ value cassandra db. again can use kafka/ spark here?

any highly appreciated.

how can upload huge csv/ tsv file (200 gb) web application? shall upload file in hdfs , specify path in ui? have split huge file small chunks (50 mb each).

depends on how web app going used. uploading file of such huge size during context of http request client server going tricky. have asynchronously. whether put in hdfs or s3 or simple sftp server matter of design choice , choice affect kinds of tools want build around file. suggest start simple ftp/nas , have needs scale, use s3. (using hdfs shared file storage haven't seen many people do, shouldn't prohibit trying)

how can unique values first column? can use kafka/ spark here? need insert timestamp/ value cassandra db. again can use kafka/ spark here?

spark batch or normal m/r job trick you. simple groupby operation, though should @ how far willing sacrifice on latency, groupby operations costly (it involves shuffles). generally, limited experience, using streaming use-cases overkill, unless continuous stream of source data. way have described use-case looks more batch candidate me.

some things focus on: how transfer file client app, end-to-end slas availability of data in cassandra, happens when there failures (do retry, etc.), how jobs run (will triggered every time user uploads file or can cron job), etc.


-

Comments

Post a Comment

Popular posts from this blog

amazon web services - S3 Pre-signed POST validate file type? -

i know it's possible put limit on file size content-length-range header. possible validate file type? https://docs.aws.amazon.com/amazons3/latest/dev/httppostforms.html#policyconditions i see there content-type header, if set say, audio/mp3 allow mp3 files , return error if file not mp3? i found previous question answers mention validating file size: s3 direct upload restricting file size , type you can specify content-type @ post request. you can specify @ signature, post must done content-type: { "expiration": "2007-12-01t12:00:00.000z", "conditions": [ {"acl": "public-read" }, {"bucket": "johnsmith" }, {"content-type: "audio/mp3"} ] } creating html form (using aws signature version 4) edit: @ previous question found, checking content-type.
Read more

c# - Check Keyboard Input Winforms -

i wondering if instead of doing this protected override void onkeydown(keyeventargs e) { if (e.keycode == keys.a) console.writeline("the key down."); } i set bool method , this: if(keydown(keys.a)) // whatever here i've been sitting here ages trying figure out how it. can't wrap head around it. in case wondering, plan call bool inside different method, check input. since want perform action after pressing key, using keydown event enough. but in cases suppose want check if specific key down in middle of process, can use getkeystate method way: [dllimport("user32.dll", charset = charset.auto, exactspelling = true)] public static extern short getkeystate(int keycode); public const int key_pressed = 0x8000; public static bool iskeydown(keys key) { return convert.toboolean(getkeystate((int)key) & key_pressed); } you should know, each time check key state using example iskeydown(keys.a) method returns true if ke...
Read more