Apache spark, 1 day

安装配置Spark

解压与权限

tar -zxvf spark-1.1.0-bin-2.4.0.2.1.4.0-632.tgz -C /opt
ln -s /opt/spark-1.1.0-bin-2.4.0.2.1.4.0-632 /opt/spark
chown -R spark:spark /opt/spark*

配置worker

vi conf/slaves

HDP125
HDP126
HDP127

注：每行一个Worker

配置spark-env.sh

cp conf/spark-env.sh.template conf/spark-env.sh

vi conf/spark-env.sh

export JAVA_HOME=/opt/jdk
export SPARK_MASTER_IP=ES122
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=FILESYSTEM -Dspark.deploy.recoveryDirectory=/opt/spark/recovery"

　　可以为Master和Worker的CPU核心和内存大小进行定制。

• SPARK_MASTER_IP：设置Master的IP
• SPARK_DAEMON_MEMORY：设置Master和Worker守护进程内存大小
• SPARK_WORKER_CORES：设置Spark应用在Worker中可以使用的CPU核数
• SPARK_WORKER_MEMORY：设置Spark应用在Worker中可以使用的内存总量
• SPARK_MASTER_WEBUI_PORT：设置Master的Web UI端口
• SPARK_WORKER_WEBUI_PORT：设置Worker的Web UI端口

　　更多配置参考：Spark Standalone配置属性

　　SPARK_DAEMON_JAVA_OPTS配置项：

System property	Meaning
spark.deploy.recoveryMode	设成FILESYSTEM来开启单节点恢复模式 ，（默认值：NONE）
spark.deploy.recoveryDirectory	Spark存储恢复状态的目录，Master能够访问

配置SPARK_HOME环境变量

vi /etc/profile

#spark
export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH
source /etc/profile

启动

启动集群

　　在master上

su spark
cd /opt/spark
sbin/start-all.sh

不用sbin脚本的方式

　　启动Worker

./bin/spark-class org.apache.spark.deploy.worker.Worker spark://IP:PORT

测试

提交Application：

bin/spark-submit --class org.hansight.spark.examples.SogouQTotal --master spark://HDP125:7077 /opt/examples-0.0.1-SNAPSHOT.jar

Final Reflection on learning

I took ES2007 firstly because I need to match the corresponding module at NTU, and also it's a indeed very useful and interesting module. Although I just need to "pass" all the modules that I have taken at NUS this semester, I spent quite some efforts in this module.

I have learnt how to use verbal and non-verbal communication skills, I have learnt the difference in culture communication and how to prepare the interview, the "STAR" strategy to answer tricky questions, thanks for Nigel.

Through my group of peer teaching, I have also learned various types of survey questions, the statistics behind them, and thus be able to conduct survey. Through the project, I learned how to do street interview, and the relevant skills in retrieving information from online sources and library.

The most memorisable skills that I have picked up are: 

1, never ignore audience even they are sitting at boundary. 

2, write thank you letter after interview.

They are included but not all. There're many useful skills and insights that I cannot enumerate all of them here.

More importantly, I got to know so many lovely classmates, although, I'm quite shy, and keep silence most of the time, I can feel all of you are friendly and warm-hearted.

All in all, I feel very lucky I have gained a lot throughout the study of this module, thanks a lot! 

Reflection on Project Proposal

My reflection on my recent project proposal regards to the treatment of migrant workers in construction industry  in Singapore.

Challenges

The challenges I faced was to put in correct materials, although we can easily get tons of information regarding the treatment of migrant workers, we need to prioritize them, select the useful information through the sea of news. Another is the street interview, I'm usually a silent man, it's challenged for me to do the street interview.

Triumphs

We successfully interviewed 1 Bangladeshi, 2 Chinese, 2 India construction workers, and we retrieved useful information throughout the simple conversation. We distribute work among the team, and everyone is contributing, and we have a integrated proposal, that's a great success.

Learning Points

I understood more about about migrant workers' current status, they are indeed mostly underpaid. And I learned how to do street interview, and how to organize information, and write the proposal.

Possible Areas for Improvement

We don't' have a large group of interview, so the sample amount is small, and we didn't try to contact MOM Singapore to get a more deep understanding of their concern and their difficulties.

It's a good opportunities for us to do this project, we learnt a lot throughout the project, and I'm becoming more appreciated for those migrant workers who indeed contribute a lot to the whole society, we will not have the air-conditioned office building without their heavy work before.

Evaluating Intercultural Behavior

Singapore is such a multi culture country, where difference people with different culture background assembled, the variety cultures bringing more funny into our daily life but unfortunately sometimes causing conflict too. 

During my third year at NTU, I stayed in HALL2. My neighbour, A, was a exchange student from Brazil, his a very funny guy, we became friend soon later. But one thing made us no longer talk to each other.

It was 10 clock at night, I decided to go to sleep earlier for tomorrow’s exam, but the loud music outside made me crazy, I knew, it was party from A again, I eventually failed to ignore the sound and shout outside:”Excuse me! you guys do have exams soon, don't you?! why not go back and take a rest?!” To my surprise, one people in the party reply to me confidently:”Hey, we are resting now!”, talk became quarrel soon. And the quarrel ended up with lose-lose situation. They stop their party disappointedly and  I became too angry to sleep well all over the night.

Few months later, I read a website, which stated that people from Brazil often party at night, especially after 10 pm. I suddenly realised at that night, I should have said something that take their culture background into more consideration. This accident actually, from then on, reminds me about the importance of choosing the correct way of dealing with people from different cultures. I should first understand other people’s culture, and identify the potential conflict, then try to avoid conflict and if necessary, solve the conflict with the culture difference in mind.

Resolving Interpersonal Conflict

I was once involved in a cross-organisation project, where my partner was a mid-aged researcher from Company A, while, I, at that time, was a year 3 undergraduate having an internship at Company B. Although we worked on the same product, we had totally different part of tasks, and yet, our parts were tightly dependent on each other.

One day, he presented his new idea of upgrading his part in a group meeting. I didn’t realise a serious problem until I went back and thought it over the night, and I then jumped up and wrote an email to him. After a long time of waiting, I got reply from him, saying he understood my concern, but, felt that it was not a big problem, and he suggested I should be able to fixed it on my side straightforward. However, I did try to work it out on my side already, unfortunately it was proven to be impossible, as his design got a serious bug that can cause an error stage, that I can do nothing to help with. I immediately told him the situation, but he then simply told me that his part was almost finished and it will be costly to change again. I tried to cc’ed my boss in the last email, but my boss was not so aware of what was difficulty, asking me to try again. I finally grew angry enough to gave up and kept silence without making any progress nor reply to their emails since then.

I was finally asked to directly explain of the problem to my boss in person, who eventually realised I was right and thus suggested my partner to make a slight change as I suggested before, but the project was therefore being delayed by 2 weeks.

Analysis: 

My partner was obviously more experienced than me feeling hat he was absolutely right, not willing to listen, and he was only concerned about his part, missing the sprite of team working.

I grew frustrated until I finally broke down the communication process, I lose to control my emotion, and I didn’t maintain the motivation. 

My boss didn't pay enough attention on my words at first, he was more concerned about solving the problem instead of solving the conflict between my partner and I, he was task-oriented instead of people-oriented.

Possible solution in view of myself:

I should calm down, control my emotion, be more self-awareness, bearing in mind that no matter how correct I was for that problem, I was just a young student in internship, I must be more patient, and explain to them more times patiently.

Question for you:

Is there any better solution if you are one of them?