Hello all,

I'm not sure either I should take DevOps IT course or Hadoop developer. I don't know much about DevOps but I heard it's market is good. On the other hand Hadoop developer seems interesting until you start learning Map reduce (programming). What do you guys recommend to get involved with? Does anyone here has experience on either DevOps or Hadoop? Honest opinions would be appreciated.

I think it soley depends on your ability and interest on those domains. They are completely different fields of IT. I do somewhat similar to DeVOps per say . I would think that DevOps is a fancy name for system admin for products usually in cloud or in virtual environments. If you have general IT experience and good understanding of software workflow, DevOps should be fairly easy to pick up. DevOps can be fascinating and incredibly challenging at the same time ...so many tools and technology to learn from.

Hadoop on the other hand I hear is complex right from the start. The things it can do is complex is used in conjunction with other complex products and to be good at hadoop you have to be good at programming ...really good. That's my thought after installing and playing with hadopp / yarn etcs.
That being said I have seen people just push jobs on hadoop for a living and make 6 figures so LUCK I guess is also an important factor.
Hadoop is fairly new and in my opinion will be replaced with some new stuff soon like for eg spark or bigquery. Mapreduce is just the tip of the iceberg...
I think of Hadoop as an ecosystem more than a technology...database / cluster / spark / storm / pig ...etc all falls under that ecosystem.


Good luck

fdpower!! Thank you for the great breakdown.

Your explanation regarding Hadoop is spot on. Its complex from day 1. While I have found HDFS, PIG, HIVE fairly learnable. Map Reduce is where I find it complex. I've spend few hours to analyze the code in one program and I bet it's not even that complex for people who have strong Java background. Lots of people in forums and reddit are saying that they don't even use Map Reduce anymore since tools like HIVE, PIG have come up as an alternative to Map Reduce. But I guess you have to master your Map Reduce skills in case those alternatives are not available when you go to a new company.

I have fairly a good background in IT including Linux, Networking. If I go for DevOps would it be a good idea?

Hi Bro...see the requirement of Hadoop .... what you want to be in hadoop .. admin or dev... admin part is easy as you have knowledge of Linux and hadoop is open source and working on Linux... like windows it has its own file system like HDFS.... I know hadoop admin part that is pretty easy... but if you want to be dev then you must have a programming ability. with hadoop lots of similar third party tool used in it... Java is good if you want to for dev... that give you a basic to understand all ...

Good luck....if you want to any other technology you can msg me...

In addition to fdpower: If you want to be a Hadoop developer, you need to know the commonly used tools/components in hadoop ecosystem. It can be fairly simple or complex at the same time to be a hadoop/big data developer and depends on which side of the hadoop ecosystem you are comfortable working with.

Consider Hadoop as a black box and lets say there is a left side of the box(1); the box itself(2); there is a right side of the box(3) & there is also a enclosure for the black box(4).

Left Side of the box(1): This is often termed as big data ingestion in the world of hadoop. The tools like Apache Sqoop, Kafka, Flume, Flafka, HDF (Hortonworks)..etc are used.

Box itself(2): Nowadays there are various open source tools like (Hive, Pig..etc) that are wrapper around old/gold MapReduce; you don't really need to write core MR code in java to execute your job. You should consider learning about the basic architecture of Hadoop, Yarn, Zookeeper, Mappers, Reducers, Partitioners, RecordReader, RecordWriter, job, job tracker etc... and also some knowledge of core java.

Right Side of the box(3): This is where you apply your business logic; Big Data analytics or Big Data Scientists, a sort of fancy name in Hadoop world. These guys treat Hadoop as a distributed storage unit. Pig & Hive are widely used for batch processing; analytics tools like Spark, HBase, Storm, Phoenix (SQL wrapper for HBase) & reporting tools like PentaHo, SAS.. etc are widely used. If you are a Phd in Physics, mathematics, even Biology, Chemistry.. you will be working in this side of Hadoop; extracting data from hadoop bridge (iPhython Notebook) or java and implement your complex machine learning, image analysis, etc.

Box Enclosure(4): This is where Dev Ops or Sys Admin role comes into play. Companies prefer preconfigured Hadoop cluster from either Cloudera or Hortonworks.. and buy subskriptions for support from these guys. Cloud service like AWS, Microsoft Azure, Rackspace..etc also provide excellent support for Cloudera or Hortonworks.

p.s. it turned out to be a long post. This is a little insight from a person who has background in Hadoop/Big Data; I hope it helps.

aaisunkaata !! Thanks for the feedback. Really appreciated.

Ruben123!! Very nice and detailed information regarding Hadoop/BigData. I really appreciate your isight.

I've been learning Hadoop for about a month now and all the materials EXCEPT writing MR code has been interesting and fun ride. I hope MR will be a smooth ride as I progress into the world of Big Data.

Can you suggest any consultancy who does placemnet for hadoop. ?

I haven't found one either. Please someone suggest if they know a consultancy that does placements in Hadoop.

interesting..
ruben123, well explained...
Can you plz fairly explain providing a real time illustration how Big data implementation benefits?
i know, It could be used everywhere..,social network, stocks, politics, hospital etc
Back in 2012, I did a POF on BigData using Horton works. The project used python skript for analyzing chunk of social network data.
Right now, we are implementing it in Stock.