orientdb2.2 - Performance tuning for loading Gigabytes of data in OrientDB -


i used etl tool insert bunch of csv data orientdb. system configuration used trial purpose ec2 m3 large ( 7.5 gib of memory, 2 vcpus, 32 gb of ssd-based local instance storage, 64-bit platform ).

the data(masked) i'm trying upload of below format :

"101.186.130.130","527225725","233 djfnsdkj","0.119836317542"  "125.143.534.148","112212983","1227 sdfsdfds","0.0465215171983"  "103.149.957.752","112364761","1121 sdfsdfds","0.0938863016658"  "103.190.245.128","785804692","6138 sdfsdfsd","0.117767539364"

the schema contains 2 node classes , 1 edge class. when tried loading data using etl tool in plocal option speed 2300 rows / second. etl configuration mentioned below : 

{    "source": { "file": { "path": "/home/ubuntu/labvolume1/orientdb/bin/0001_part_00" } }, "extractor": { "csv": {"columnsonfirstline": false, "columns":["ip:string", "dpcb:string", "address:string", "prob:string"] } },   "transformers": [ { "merge": { "joinfieldname":"ip", "lookup":"ipaddress.ip" } }, { "field":   { "fieldname": "addr_key",      "expression": "dpcb.append('_').append(address)" } },{ "vertex": { "class": "ipaddress" } }, { "edge": { "class": "located",             "joinfieldname": "addr_key",             "lookup": "phylocation.loc",             "direction": "out",             "targetvertexfields": { "geo_address": "${input.address}", "dpcb_number": "${input.dpcb}"},             "edgefields": { "confidence": "${input.prob}" },             "unresolvedlinkaction": "create"         }     } ], "loader": { "orientdb": {    "dburl": "plocal:/home/ubuntu/labvolume1/orientdb/databases/bulk_transfer_test1",    "dbtype": "graph",    "dbuser": "admin",    "dbpassword": "admin",    "serveruser": "admin",    "wal": false,    "serverpassword":"admin",    "classes": [      {"name": "ipaddress", "extends": "v"},      {"name": "phylocation", "extends": "v"},      {"name": "located", "extends": "e"}    ], "indexes": [      {"class":"ipaddress", "fields":["ip:string"], "type":"unique" },      {"class":"phylocation", "fields":["loc:string"], "type":"unique" }    ] }   } }

then separated vertices files , ran etl job vertices, time speed close 12500 rows / second. reasonably fast , kind of works me. ( when removed indexes speed doubled) config used :

{   "source": { "file": { "path": "/home/ubuntu/labvolume1/orientdb/bin/only_ip_05.csv" } },   "extractor": { "csv": {"columnsonfirstline": false, "columns":["ip:string"] } },   "transformers": [ { "vertex": { "class": "ipaddress" } }],   "loader": { "orientdb": {    "dburl": "plocal:/home/ubuntu/labvolume1/orientdb/databases/bulk_transfer_test7",    "dbtype": "graph",    "dbuser": "admin",    "dbpassword": "admin",    "serveruser": "admin",    "wal": false,    "serverpassword":"admin",    "classes": [      {"name": "ipaddress", "extends": "v"}    ],    "indexes": [      {"class":"ipaddress", "fields":["ip:string"], "type":"unique" }    ] } } }

however when tried insert edges alone speed became extremely slow @ 2200 rows / second. turned out lower running entire operation within 1 run. config file attached below :

{   "source": { "file": { "path": "/home/ubuntu/labvolume1/orientdb/bin/edge5.csv" } },   "extractor": { "csv": {"columnsonfirstline": false, "columns":["ip:string", "loc:string", "prob:string"] } },   "transformers": [ { "merge": { "joinfieldname":"ip", "lookup":"ipaddress.ip" } }, { "vertex": { "class" : "ipaddress", "skipduplicates" : true }}, { "edge": { "class": "located",             "joinfieldname": "loc",             "lookup": "phylocation.loc",             "direction": "out",             "edgefields": { "confidence": "${input.prob}" },             "unresolvedlinkaction": "nothing"         }     }  ],   "loader": { "orientdb": {    "dburl": "plocal:/home/ubuntu/labvolume1/orientdb/databases/bulk_transfer_test7",    "dbtype": "graph",    "dbuser": "admin",    "dbpassword": "admin",    "serveruser": "admin",    "wal": false,    "tx":false,    "batchcommit":10000,    "serverpassword":"admin",    "classes": [      {"name": "ipaddress", "extends": "v"},      {"name": "phylocation", "extends": "v"},      {"name": "located", "extends": "e"}    ] }   } }

please can let me know if i'm doing wrong here, please suggest better ways performance improvement


-

Comments

Post a Comment

Popular posts from this blog

amazon web services - S3 Pre-signed POST validate file type? -

i know it's possible put limit on file size content-length-range header. possible validate file type? https://docs.aws.amazon.com/amazons3/latest/dev/httppostforms.html#policyconditions i see there content-type header, if set say, audio/mp3 allow mp3 files , return error if file not mp3? i found previous question answers mention validating file size: s3 direct upload restricting file size , type you can specify content-type @ post request. you can specify @ signature, post must done content-type: { "expiration": "2007-12-01t12:00:00.000z", "conditions": [ {"acl": "public-read" }, {"bucket": "johnsmith" }, {"content-type: "audio/mp3"} ] } creating html form (using aws signature version 4) edit: @ previous question found, checking content-type.
Read more

c# - Check Keyboard Input Winforms -

i wondering if instead of doing this protected override void onkeydown(keyeventargs e) { if (e.keycode == keys.a) console.writeline("the key down."); } i set bool method , this: if(keydown(keys.a)) // whatever here i've been sitting here ages trying figure out how it. can't wrap head around it. in case wondering, plan call bool inside different method, check input. since want perform action after pressing key, using keydown event enough. but in cases suppose want check if specific key down in middle of process, can use getkeystate method way: [dllimport("user32.dll", charset = charset.auto, exactspelling = true)] public static extern short getkeystate(int keycode); public const int key_pressed = 0x8000; public static bool iskeydown(keys key) { return convert.toboolean(getkeystate((int)key) & key_pressed); } you should know, each time check key state using example iskeydown(keys.a) method returns true if ke...
Read more