In a Hadoop setup (rather any Big Data setup), memory issues are not unexpected!
An update on couple of issues we have seen off late –
1. NameNode process gets stuck:
In this case, typically you will see following symptoms –
a. DataNode gives following timeout error -
WARN ipc.Client: Exception encountered while connecting to the server : java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/<datanode-ip>:<datanode-port> remote=/<namenode-ip>:<namenode-port>]
ls: Failed on local exception: java.io.IOException: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/<datanode-ip>:<datanode-port> remote=/<namenode-ip>:<namenode-port>]; Host Details : local host is: "<datanode-ip>"; destination host is: "<namenode-ip>":<namenode-port>;
ð What this essentially means is that DataNode process timed-out while trying to connect to the NameNode. So obviously the next step is to check why NameNode didn’t respond.
b. On checking NameNode logs, we observed following warning –
WARN org.apache.hadoop.util.JvmPauseMonitor (org.apache.hadoop.util.JvmPauseMonitor$Monitor@18df3f69): Detected pause in JVM or host machine (eg GC): pause of approximately 74135ms No GCs detected
This indicates that the NameNode paused for longer than expected time of 60000ms. This also explains why DataNode did not get response from NameNode in designated 60000ms.
The warning also indicates that the pause was not due to GC. Typically GC can cause such ‘stop the world’ pauses and if that’s the case, it calls for a memory profiling and GC Tuning.
However, in this case, it turned out CPU activity was very high on the master node due to another cronjob. We sorted out the cronjob issue and the issue was resolved.
2. DataNode process OOM:
Depending on the size of data and amount of data activity, you may observe OOM issue in DataNode process once in a while.
A quick fix would be to allocate more memory to DataNode process. Typically following configuration change will be helpful –
Update value of HADOOP_DATANODE_HEAPSIZE in <HADOOP HOME>/conf/hadoop-env.sh
Also, it is advisable to configure data-node to generate heap-dump on OOM error. That will help you with further analysis of heap if you get same error again.
(This is applicable to other processes as well – NN/RM/NM etc.)
Regards,
Sarang
Hi Sarang,
ReplyDeleteWARN org.apache.hadoop.util.JvmPauseMonitor (org.apache.hadoop.util.JvmPauseMonitor$Monitor@18df3f69): Detected pause in JVM or host machine (eg GC): pause of approximately 74135ms No GCs detected
Can you tell me the property name used to change the time duration (which here is 74135ms) of namenode ?
My dad was diagnosed with Alzheimer's diseases this neuron disease started 25 years before being revealed it was triggered the moment we started eating sugars including any and almost all carbs that break down to glycerol,and he was placed on the best medical treatment approved by united states,(FDA) he has been taking Cholinesterase,Memantine to treat his memory loss,confusion and the problems with his thinking and reasoning.just to help slow the progression and manage the symptoms,as no cure exists I read through with interest. I once believed Alzheimer disease has no cure. Well it is true with English medicine, but not with herbal medicine. My Dad's experience opened my eyes to the reality of a cure through herbs. My Father was a vibrant man before his encounter with Alzheimer. He was a scientist, hence his mental capability was never in doubt. In 2013, he was diagnosed with Alzheimer. The symptoms manifested through repeating conversation and gradually forgetting things. It became progressive from finding the right words during conversation, to significant confusion and forgetting details about himself. It was not a good experience, seeing your father whose brilliance had no match, totally became a shadow of himself. His doctors said the disease has no cure, just medications for treatment which had a lot of side effects were administered to him. Early 2016, while on the internet, I bumped into a story of Alzheimer, and I read about a cure through herbs with interest. I researched more to be sure it was not a hoax. In my quest, Doctor James herbal mix medicine was mentioned in the testimony. I contacted him on his email that was provided (drjamesherbalmix@gmail.com) and I got his herbal mix medicine for my father.He told me that his herbal mix medicine will help my Dad to reduce the abnormal protein fold inside his neurons,and regulate the nutrients and molecules in his body system,and as well stop the progressive disorder that build up in damaging his brain cells,and help his weak cells that causes brain shrinking to function well,it's a good herbal drinks for cell repairing.This doctor James is super great man and his herbal mix medicine is wonderful and works effectively as he said,with no side effects. It's been 4 years and my Dad is perfectly okay and back to his laboratory work even at old age. For your loved ones with Alzheimer or Dementia, take them off English medicine and use Dr. James herbal mix medicine for treatment. if you have any kinds of health challenges or diseases such as..Parkinson's disease,Schizophrenia,Lung Cancer,Breast Cancer,Colo-Rectal Cancer,Blood Cancer,Prostate Cancer ,Epilepsy Dupuytren's disease ,Coeliac disease,Creutzfeldt–Jakob disease,Cerebral Amyloid Angiopathy, Ataxia,Arthritis,Amyotrophic Lateral Sclerosis,Fibromyalgia,Fluoroquinolone Toxicity
ReplyDeleteSyndrome Fibrodysplasia Ossificans ProgresS sclerosis,Seizures,Alzheimer's disease,Adrenocortical carcinoma.Asthma,Allergic diseases ,Copd,Glaucoma., Cataracts,Macular degeneration,Cardiovascular disease,Lung disease.Enlarged prostate,Osteoporosis,Lupus,Cushing’s disease,Heart failure,Multiple Sclerosis,Hypertension,Lyme Disease,Blood Cancer,Brain Cancer,Breast Cancer,Lung Cancer,Kidney Cancer, HIV/AIDS, Herpes Virus,Hepatitis B, Liver Inflammatory,Diabetes,Fibroid. just reach him on his Email [drjamesherbalmix@gmail.com]. and get your permanent cure.he's a good man and he will help you.