hadoop - TestDFSIO benchmarking on cdh 5.8.0 -


environment details:

os : centos 7.2 cdh: cdh 5.8.0 hosts: 11 ( 2 masters, 4 dn+nm, 5 nm)

yarn.nodemanager.resource.memory-mb 32074mb (for nodemanager group1) 82384mb (for nodemanager group2)

i have hadoop cluster 11 nodes, 2 masters, 4 slaves datanode & nodemanager daemons running, 5 nodes nodemanager daemon running on them. on cluster, running testdfsio benchmarking job 8tb load having 10000 files , file size of 800mb each. have noticed few things not understand properly.

1) number of splits job shown 10000. how come 10000 splits, dfs.blocksize shows 128mb, going setting, number of splits should more 10000 right?

2) in resoucemanager web ui, saw on 5 computenodes ( nodes on nodemanager running) 32 map tasks have run on each of these nodes. other map tasks being run on 4 dn+nm nodes. why happening? have allocated 9 slave nodes 2 node groups. 4 dn+nm nodes in nodemanager group1 , other 5 slaves in nodemanager group2. yarn.nodemanager.resource.memory-mb slaves in nodemanager group1 32074mb , slaves in nodemanager group2 82384mb. think ideally, 5 slave nodes in nodemanager group2 should take more map taks. why not happening?

  1. afair - testdfsio allocated map task per file. why end same number of map tasks, tho block size smaller.

  2. how data locality configured? mappers prefer nodes data local. explain why more tasks on nodes datanodes being local.


-

Comments

Post a Comment

Popular posts from this blog

amazon web services - S3 Pre-signed POST validate file type? -

i know it's possible put limit on file size content-length-range header. possible validate file type? https://docs.aws.amazon.com/amazons3/latest/dev/httppostforms.html#policyconditions i see there content-type header, if set say, audio/mp3 allow mp3 files , return error if file not mp3? i found previous question answers mention validating file size: s3 direct upload restricting file size , type you can specify content-type @ post request. you can specify @ signature, post must done content-type: { "expiration": "2007-12-01t12:00:00.000z", "conditions": [ {"acl": "public-read" }, {"bucket": "johnsmith" }, {"content-type: "audio/mp3"} ] } creating html form (using aws signature version 4) edit: @ previous question found, checking content-type.
Read more

c# - Check Keyboard Input Winforms -

i wondering if instead of doing this protected override void onkeydown(keyeventargs e) { if (e.keycode == keys.a) console.writeline("the key down."); } i set bool method , this: if(keydown(keys.a)) // whatever here i've been sitting here ages trying figure out how it. can't wrap head around it. in case wondering, plan call bool inside different method, check input. since want perform action after pressing key, using keydown event enough. but in cases suppose want check if specific key down in middle of process, can use getkeystate method way: [dllimport("user32.dll", charset = charset.auto, exactspelling = true)] public static extern short getkeystate(int keycode); public const int key_pressed = 0x8000; public static bool iskeydown(keys key) { return convert.toboolean(getkeystate((int)key) & key_pressed); } you should know, each time check key state using example iskeydown(keys.a) method returns true if ke...
Read more