Saturday, 6 April 2019

Java Top 100 Interview question and answer

Q:- Wrapper Classes in java?
Ans:-Wrapper classes are used for converting primitive data types into objects,
like int to Integer
here are mainly two uses with wrapper classes.

1) To convert simple data types into objects
2) To convert strings into data types


The eight primitive data types byte, short, int, long, float, double, char and boolean.

Why we need wrapper class?
For example: While working with collections in Java,
we use generics for type safety like this: ArrayList<Integer> instead of this ArrayList<int>.
The Integer is a wrapper class of int primitive type.
We use wrapper class in this case because generics needs objects not primitives.


Wrapper class Example: Primitive to Wrapper



2. Wrapper class objects allow null values while primitive data type doesn’t allow it.



2Example1:-
class WrapperClass{
psvm(){

int i=5; //primitive data type
Interger li= new Integer(i); // so puting value insdie object is called Boxing

int j = li.intValue(); // unboxing


---------------------------------------------------------------------------------

Q: STring buffer and Stringbuilder?
Ans: StringBuffer is syncronised an thread  safe but StringBuilder is not synronised an not thread safe
StringBuffer and StringBuiler both are mutable

Example:

String str= new String("Hello");
StringBuffer buffer= new StringBuffer("Hello");
StringBuilder builder = new StringBuilder("Hello");

str.concat("Hello");
buffer.append("Hi");
builder.append("Hi");

System.out.println(str);
System.out.println(buffer)
System.out.println(builder);

OUTPUT:-
Hello  ->  here string is imutable that why its not changable  so it will take first string  data
Hi     -> here it is mutable means its  changeable that why append ouput came 
Hi    -> here also it is mutable so append out put came

here

-------------------------------------------------------------------
Q: diff btn == and equal()?

== method is cpmpare reference
equals() method its compare content of the object

example:

String str1= new String("Hello");
String str2 = new String("Hello");

if(str1 == str2){ // comparing reference
  system.out.println("str1== str2");
}else{
   system.out.println("str1  !== str2");
}

if(str.equals(str2)){
  system.out.println("str1 is equal str2");
}else{
    system.out.println("str1 not equals str2");
}

OUTPUT:-
str1 !== str2
str1 is equal str2
----------------------------------------
Q: Diff btn Abstract class and interface?

Abstarct class vs Interface.
1)An abstarct class  can provide complete default code
and/or just the details that have to be overriden.
where as interface can not provide any code at all, just the signature

2)in case of abstarct class, class may extend only one abstarct class
where as in interface A class may implement serveral interface.

3)abstarct class can have not abstract method.
where as interface can have all methods of interface are abstract

4)abstract class can have instance variable
where as An interface can not have instance variables.

5)abstract class can have any visisbilty: public, private, protected
where as interface visibility must be public or none

6)abstarct class can contain constructor
where as interface can not conatin constructors.

7)abstract classes are fast
where as interfce slow as it required extra indirection to find corresponding method in the actual class.


------------------------------------------------------------------------------
Q:What is overriding with example?
ans: declare the method in sub class which is already present in superclass is called overriding.

we can say method name , signature and return all should be same

-------------------------------------------------------------------------------
Exmaple: What will be the output of this program?

class Parent
{
  public static void foo(){
  System.out.println("i am foo in parent");
 }


 public  void bar(){
  System.out.println("i am bar in parent");
 }

}

class Child extends Parent{

   public static void foo(){System.out.println("i am foo in child
   
 }

public  void bar(){
  System.out.println("i am bar in child");
 }

public static void main(String args[])
{

Parent par = new Child();
Child child = new Child();

par.foo();
child.foo();

par.bar();
child.bar();

}

}


Output:

i am foo in super
i am foo in child
i am bar in child
i am bar in child

------------------------------------
Q:Servlet lifecycle?
ans:-there are five stages.
>servlet is loaded
>servlet is instantiated
>servlet is initialized
>service the request
>servlet is destroy.
-------------------------------------------------------------------------------------------
Q:- What is Request Dispature?
RequestDispature  interface is used to forward the request to another resource
that can be html,jsp or anothr servlet in same application


we can also use include content of another resource to the response
there are two method defines in this interface:
1) forward()
2) include()
----------------------------------------------------------------------------------------
Q:- What is bean in spring and Explain scops of bean in Spring?
A bean is an object that is instantiated, assembled and managed by sping IoC container.
They are managed by the Spring IOC COntainer.
There are 5 scops of bean spring
singleton
prototype
request
session
global-session
-----------------------------------------------------------------------------------------
Q:- What is dispature servlet and COntextLoaderListener?
DispatcherServlet:-
DispatcherSrevlet is front controller in spring mvc
application as it loads the spring bean configruation file
and initializes all the bean that have been configured

ContextLoaderListener:
ContextLoaderListener , the listener to start up and shut don the webapplicationCOntext in spring root.
------------------------------------------------------------------------------------------
Q: diff btn COnstructor injection and setter injection?

ANs:-
COnstructor injection:-                        setter injection:
1-no partial injection                         1-Partial injection   

2-Doesnt override the  setter property         2-Overrides the constructor property if both are difined.   

3-Create new instance if any modification      3- DOesnt create the  new instance if you change the property value
occurs.

4- Better for too may properties.              4- better for few properties.


----------------------------------------------------------------------------------------
QA:- What is autowiring in spring? what are the autowiring modes?
Ans"- Autowiring enables the programmer to inject the bean automatically.
We dont need to write explicit injection logic.
example:
<bean id="emp" class="com.java.Employee" autowire="byName"/>

There are 4 type of autowiring?
1- no:- This is default mode. it means autowiring is not enabled.

2-byName:- injection the bean based on the property name.it uses setter method

3- byType:- injection the bean on the property type. it sues setter method.

4- constructor:- it injects the bean using constructor.

-----------------------------------------------------------------------------------------
Q:- How to handle exception in spring mvc framwork?
Ans:- Spring MVC provide some way to help us acchive exception
1- Controller Based: we can define excepton handle methods in our contrller class
2- Global Exception Handle: Exception Handling is a cross cutting concern and spring provides
3- HandlerExcptionResolver
----------------------------------------------------------------------------------------
Q:- What are some of the important Spring annotation used?
Ans:-
@Controller
@PathVariable
@Qualifier
@Configruation
@Scope
@RequestMapping
@AutoWired
@Service
@Aspect
------------------------------------------------------------------------------------------
Q:- How to integarte spring and hibernate framwork?
Ans:- we can use Spring ORM module to integtae Spring and hibernate framwork.
Spring ORM also provide support for using Spring declarative transaction management.
-----------------------------------------------------------------------------------------

Q:- What is Hibernate?
Ans:- Hibernate is java based ORM tool that provides framework for mapping application domain onjects to the relational database tables and vice versa.

Hibernate provide reference implementaion of java Persistence API(JPA).
That makes its great choice as ORM tool with benefits of loose coupling
loose coupling means where dependency level reduce to maximum level.
Hibernate configruation are flexible and can be done from Xml configruation file as well as programmatically

-------------------------------------------------------------------------------------------------

Q:- diff btn get and load method in hibernate?
ans:Both are used to get the data from the database
get()-                                   load():
1- Returns null if obejct is not found   1- ThrowsObjectFoundException if object is not found.
2- get() method always hit the database  2- load() method doesnot hit the database, first it will search in cache memory not there than it will hit database.
if data is not there it will retrun null      if data is not there it will throw exception ObjectNotFOund
3- it return real object not proxy       3- it returns proxy object.
4- it should used if you are not sure    4- it should be use if you are sure  that instance.
    about the existence of instance.
==================================================================================

Q:- What is syncronizantion in the thread?
Ans:-syncrinizantion means - it is the process of enableing the lock of object
every java object is associcated with some lock mechanism , by default this lock is disable.
if you want to enable this lock so that it should work in multithreade enviroment.
multiple thread should not enter in critical section at once so there we can use syncronism.

just to enable of lock of object so that multiple thread should not allow to access concurrently.
----------------------------------------------------------------------------------------
Q:- diff btn runnable and thtead?
ans:-1-runnable is interface,  availble in java.lang package
it has only one absract method.. public void run()
 where as thread is concreat class availble in java.lang package, it is also implementing runnable interface
it has thread class has overwritten the run method they have given dumy implementation.
-----------------------------------------------------------------------------------
Q:Which one is better runnable or thread?
Ans:-its depend up on requirement

both implementation is same
only diffrence when you are writting your user define  thread class by extending thread
bcoz java doen not support multiple inheritance with class's so you can not extend any other class
bu when you are writting user define thread by implementing runnable interface so what will have happen
you have change to extend other class also you can implement other interface.

-----------------------------------------------------------------------------------------
Q:- What is auto boxing?
ans:-auto boxing is feature of java 5
you can not store primitive value into collection,

so when ever you have requirement to get primitive as a object type u have to use wrapper class.
you have to create object of Integer class u have to pass primitive than u can get wrapper

Java 5 support autoboxing and unboxing feature

AutoBoxing:-autoboxing is proceess of converting primitive to wrapper and wrapper to primitive automatically.

--------------------------------------------------------------------------------------------
Q:-diff java 7 and java 8?
java 7 they have added some new feature
1: in one catch block we can write multiple exception
2:- they have gievn rethroughing execption imporived type checking
3-dymond operator
4-they have given try with resource

java8:- they have added new feature
1:- lamda expresion and functional interface
2-Stream API
3-forEach() in iterable interface
4-collection
5-Date and time Api

Q:- Expalin java 1.8 feature?
Ans:-
1)in java 1.8 one of the best feature they have given functional programming
that you can achiev by usinh lamda expression so thats help you to reduce lot of code.
Boilerplate  code you will reduce by using lamda expresion

Boilerplate code means a piece of code which can be used over and over again. On the other hand,
anyone can say that it's a piece of reusable code.

2) They have given java streaming API means
you can write some application which will help in parrell processing.

3) Next they have given utility classes like collection class's , you have array class's
that has only predefine method so

-------------------------------------------------------------------------------------------
Q:- What is diffrence between Soap base web services  and Rest based web services?
Ans:-
1-SOAP based web service- it is based on some standard they are follwing some standard like interface hirarchy is there
But in Rest based web service there is no starbdard

2-Soap based web services you have to use the protocal SOAP/Http
where as Rest full web srvice only based on Http protocal.

3-SOAP based web service will fouce you to use only Xml for communcation but in RESt be]ased web service
you have flexibility, you can use json , you can use xml, you can use simple text, Html and pdf

4- SOAP web services are very havvy u must have to use IDE to develop that
BUt rest full  web service even you can develop in notpad its very light weight

5- To test the SOAP based web service its a long process but To test the rest based web service
you can test using simple url also.

--------------------------------------------------------------------------------------------
Q:-What is servletContext object?
ans: servletContext object is..
It is the web container object like  three type of scope you have in servlet
servletContext , servlet request and session
SO if you want to use any data  among ll servlet and all the user can use  that data
so that you can write the web.xml file servletContext parameter you can define
that is universal object that you can access any where from the application to the end user.

---------------------------------------------------------------------------------------------
Q:- how many instnace of serveltContext will be there?
Ans:- only one

Q:- What is servletCOnfig?
Ans:- servletConfig object it is related to your particular servlet
so if you have requiremnet  to use some data for  particular servlet
how many user comming that many object will be created.

-------------------------------------------------------------------------------------------

Q:- Diff btn HashSet and linkedHashSet?
Ans:
in HashSet data will store based on hashCode value of object you can call that random order
Random order means what ever order you are stroing    that order it will no be store
And HashSet internally using index representation so searching will be fast.

LinkedHashSet data will be store in the same order added by the user  and linkedHashSet internally using
node data structure so when you have requirement perform insert , delete operation
 than we will go for linkedHashSet .
------------------------------------------------------------------------------------
Q:- is there any collection that where can store key value paire and key has to be in sorted  order?

Ans:-TreeMap
suppose if i add Employee object in treeMap so what will  happen?
ANs;- it will give classCastException
so if you want use Employee object so key must be comprable object
when ever you inserting any key inside the treeMap that key must be comprable.
So Employee class has to implement comprable interface, it has override compareTo method
in that you have to return any field like employee id or employee name.
so it will give you the nature sorting order.

----------------------------------------------------------------------------------

Q:- What is diffrence between wait() and sleep()?
Ans:-
Wait() is instance method of java object class, java.lang Object class
It has three overloaded method 
wait() with witout any parameter
wait(long timeout) with long mili second
wait(long timeout, int nanos)  with long and integer

Sleep()- sleep is a static method of thread class
this method is  belong to thread class
it has only two overloaded method
sleep(long mili) with long parameter
sleep(long mili, int nanos) with long, int parameter


 Sleep() method is used to pause the execution of thread for specify demon of time

Main diffrence
when we are using in syncronized context
suppose your method is syncronized in that run() method , whatever method
if you are calling sleep() method or run() method than atucall diffrnce will come.

what will happen when you call sleep() method on running thread and than thread will go to sleep state
it will not lock of object but in the case of wait() method thread will go to wait() state
and emidailty it will release lock of object,
SO that is why wait() method is responsible to release the loack of an object
that is why we have to call wait() method from syncronised context only
BUt sleep() method can call non syncronised also 


Q- Diffrence btn HashMap and concurrentHashMap? why it is actually comes in picture?
Ans:- first i need to compare hashMap and hashTable
hashTable is syncronised 
so thats means, at a time one thread can access the data from the hashTable
it can write or read that is hashTabel.
But hashmap it is not syncronised means mutiple thread its support concurrence
Now in hashtable the problem  was..like when ever thread writing to it or reading it
it will lock entire object entire hashMap object so it will not allow any other
suppose hashMap having 1 lakh entry so only one thread can access other thread will not allow to access

Apache Spark Interview Question

    Top most 50 Spark interview question
---------------------------------------------------------

Q:-How do you connect Hadoop in your project?
ans:-Through Edge Node we can connect to hadoop cluster.

Q:-What is spark?
ans:-spark is open source cluster computing framework
it does real time data processing as well as batch processing also
but hadoop can do  batch processing only.

-------------------------------------------------------------------------------------
Q:- Why spark? why not hadoop?
Ans:-There are several reason
1-main reason hadoop can do batch processing only but spark can both real time data processing and
batch processing.

2-No of line of code which you write in spark is less compare to hadoop.

3-Hadoop is written in java but spark is written in scala.
scala is language which is written in scala.


--------------------------------------------------------------------------------------

Q:-What are the feature of spark?
ans:-
1- real- time stream processing:- spark offer real time processing data but
Hadoop MapReduce was able to handle and process data which is already present but
it does not support real-time data processing thats why spark comes in picture.

2- dynamic in nature:- easily possible to develop a parallel application.

3-in memeory computation in spark:-no need to fatch data every time from disk
4- Reusability:-
5-Fault Tolerance :- apche spark provide fault tolerance through spark abstraction-RDD ,
Spark RDD are capable to digonosis the failure of any worker node in the cluster.

6- Lazy Evaluation-lazy evaluation means that execuation will not start util an action is triggered
Transformation are lazy in nature means when we call some operation in RDD, it does not execute immediatly.

------------------------------------------------------------------------------------------------------
Q:-Advantage of spark?
ans:- 1-increase Managability
2- saves computation and increass speed
3-reduce complexities
4-optimization.
----------------------------------------------------------------------------------------------------
Q:-What are the component of spark Ecosystem?
ans:-Spark core
spark streaming
spark sql
Graphx
MLIB

----------------------------------------------------------------------------------------------------
Q:- Lifecycle of spark?
ans:-
Lifecycle:-
1-load data on cluster
2-create RDD(once u creat RDD yu have to transformation)
3-DO transformation
4-Perfrom action
5-create data fram
6- perform query on dataFram.


another word-

1- initial part loading the data
you can load streaming data.

2-once data is loaded we need transform the data
using the transformation we use map, flat map, filter

3-once transformation is complete we have to perform action of the transform data
----------------------------------------------------------------------------------------------

Lifecycle of Spark program
-------------------------------------------------------------------------------------------------
Q:-The following steps explain the lifecycle of a Spark application with standalone resource manager,
and Figure 3.8 shows the scheduling process of a spark program:
Ans:-
1-The user submits a spark application using the spark-submit command.

2-Spark-submit launches the driver program on the same node in (client mode) or on the cluster (cluster mode)
and invokes the main method specified by the user.

3-The driver program contacts the cluster manager to ask for resources to launch executor JVMs
based on the configuration parameters supplied.

4-The cluster manager launches executor JVMs on worker nodes.

5-The driver process scans through the user application.
Based on the RDD actions and transformations in the program, Spark creates an operator graph.

6-When an action (such as collect) is called, the graph is submitted to a DAG scheduler.
The DAG scheduler divides the operator graph into stages.

7-A stage comprises tasks based on partitions of the input data. The DAG scheduler pipelines operators...


in other word:-
1- spark has something called driver
driver is like master that is the one which is going to command to every one

2-inside the driver we have something called spark context it just like spring context.

3-so this spark context is going to controll worker nodes
inside the worker nodes we have executors node these executor basically will execute some task.

4-so there is driver which act as a master, Driver instruct the worker to execute some task
that is done by spark context so spark context instruct to worker to execute some task on each node

5- if particular task is failed than it get rebuild and spark context rebuild and sent to worker again.

--------------------------------------------------------------------------------------------------

Q:- Are there any benifit of Apache Spark over HAdoop MApreduce?
Ans:- Spark has ability to perform data processing 100 times faster than Mapreduce
Also SPark has inbuilt memory processing and libraries to perform multiple task together
like batch processing, streaming. intractive processing etc..


Q:-Define Apache spark core and how it is useful for scala developer?
Spark core is used for memory management, job monitoring, tolerate faults
scheduling jobs and interactive storage feature. Rdd is advance feature in spark core suitable for tolerating fault
RDD is a collection of distributed objects available across multiple nodes
that are generally manipulated in parallel.


Q:-HOw many cluster mode are supported in Apache Spark?
Ans:-
There are three cluster mode are supported
standalone
mesos
YARN cluster managers

Q:- What are the way to launch spark over YARN?
Ans:- when we running on YARN spark executors  runs as YARNS Container
spark support two mode for running on YARN

1- cluster mode:- it means it will be use full in production envirement by
 using spark-submit or spark-shell--master yarn-cluster command

2-client mode:- it is useful for development purpose
bu using this command :- spark-submit or spark-shell -- master yarn-client.


Q:- WHile submit the spark job what are the properties you are going to submit?
ans:


Q:-What is SparkContext in Apache Spark?
ans:SparkContext is the entry point of Spark functionality.
The most important step of any Spark driver application is to generate SparkContext.
 It allows your Spark Application to access Spark Cluster with the help of Resource Manager.
The resource manager can be one of these three- Spark Standalone,  YARN, Apache Mesos.

----------------------------------------------------------------------------------------------------

Q:- Who will create  sparkCOntext?
ans:-SparkContext is the entry gate of Apache Spark functionality.
The most important step of any Spark driver application is to generate SparkContext.
 It allows your Spark Application to access Spark Cluster with the help of Resource Manager.

-----------------------------------------------------------------------------------------------------

Q:How to Create SparkContext Class?
ans:-If you want to create SparkContext, first SparkConf should be made.
The SparkConf has a configuration parameter that our Spark driver application will pass to SparkContext.

or

In short, it guides how to access the Spark cluster. After the creation of a SparkContext object,
 we can invoke functions such as textFile, sequenceFile, parallelize etc.
The different contexts in which it can run are local, yarn-client, Mesos URL and Spark URL.
Once the SparkContext is created, it can be used to create RDDs, broadcast variable,
and accumulator, ingress Spark service and run jobs.
All these things can be carried out until SparkContext is stopped.

------------------------------------------------------------------------------------------------------
Q:-What are Stages in Spark?
Ans:-A stage is nothing but a step in a physical execution plan. Moreover,
It is a physical unit of the execution plan. In other words, Stage is a set of parallel tasks
i.e. one task per partition. Basically,
each job which gets divided into smaller sets of tasks is a stage.

3. Types of Spark Stages
Basically, stages in Apache spark are two categories

a. ShuffleMapStage in Spark

b. ResultStage in Spark

-----------------------------------------------------------------------------------------------------
Q:What is Spark Executor?
ans:-Basically, we can say Executors in Spark are worker nodes.
Those help to process in charge of running individual tasks in a given Spark job.
Executors also provide in-memory storage for Spark RDDs

--------------------------------------------------------------------------------------------------------
Q:-  Can we run Apache Spark without Hadoop?
ans:-Yes, Apache Spark can run without Hadoop,

standalone, or in the cloud. Spark doesn’t need a Hadoop cluster to work.
Spark can read and then process data from other file systems as well.HDFS is just one of the file systems that Spark supports.

Spark is a meant for distributed computing. In this case, the data is distributed across the computers and
Hadoop’s distributed file system HDFS is used to store data that does not fit in memory.


-------------------------------------------------------------------------------------------------------
Q:-Different Running Modes of Apache Spark?
Ans:-Apache Spark can be run in following three mode :

(1) Local mode
(2) Standalone mode
(3) Cluster mode

----------------------------------------------------------------------------------------------------------
Q:-What are the roles and responsibilities of worker nodes in the apache spark cluster?
Is Worker Node in Spark is same as Slave Node?

Ans:-Worker node refers to node which runs the application code in the cluster.
 Worker Node is the Slave Node. Master node assign work and worker node actually perform the assigned tasks.
 Worker node processes the data stored on the node,
they report the resources to the master. Based on the resource availability Master schedule tasks.

-----------------------------------------------------------------------------------------------------------

Q:-what are the features of dataframe in Spark?
List out the characteristics of DataFrame in Apache Spark.

Ans:-DataFrames are the distributed collection of data. In DataFrame, data is organized into named columns.
It is conceptually similar to a table in a relational database.

Out of the box, DataFrame supports reading data from the most popular formats, including JSON files,
Parquet files, Hive tables. Also, can read from distributed file systems (HDFS), local file systems,
cloud storage (S3), and external relational database systems through JDBC.
----------------------------------------------------------------------------------------------------

Q:-What are the different methods to run Spark over Apache Hadoop?
Ans:-
1. Local standalone mode — everything (spark , driver , worker , etc) are on the same machine locally.
Generally used for testing and develop the logic for spark application

2. YARN client — the client which runs driver program and submits the job is same ,
 and the workers (data nodes) are separate.

3. Yarn Cluster — the driver program runs on one of the dedicated data nodes and
the workers are separate. Most advisable for production platform.

3. Mesos:

Mesos is used in large scala production deploymen


-------------------------------------------------------------------------------------------------
Q:-What is Parquet file format ? Where Parquet format should be used ? how to convert data to Parquet format ?
Ans:-
Parquet is the columnar information illustration that is that the best choice
for storing long run massive information for analytics functions.
It will perform each scan and write operations with Parquet file.
Parquet could be a columnar information storage format.

Parquet is created to urge the benefits of compressed,
economical columnar information illustration accessible to any project,
 despite the selection of knowledge process framework, data model, or programming language.

Parquet could be a format which will be processed by variety of various systems: Spark-SQL,
 Impala, Hive, Pig, niggard etc. It doesn’t lock into a particular programming language
since the format is outlined exploitation, Thrift that supports numbers of programming languages.
as an example, Aepyceros melampus is written in
C++ whereas Hive is written in Java however they will simply interoperate on an equivalent Parquet information.


----------------------------------------------------------------------------------------------------

Q:- What is RDD?
ans:-
RDD stands for resilient distributed dataSet
RDD is immutable fault tolrance distributed object run as parallel
      or

RDDs are immutable(can’t be modified once created) and fault tolerant, Distributed because
it is distributed across cluster and Dataset because it holds data.

         or

RDD is a collection of distributed objects available across multiple nodes
that are generally manipulated in parallel.



So why RDD? Apache Spark lets you treat your input files almost like any other variable,
 which you cannot do in Hadoop MapReduce.
RDDs are automatically distributed across the network by means of Partitions.



Partitions

RDDs are divided into smaller chunks called Partitions, and when you execute some action,
a task is launched per partition. So it means, the more the number of partitions, the more the parallelism.
Spark automatically decides the number of partitions that an RDD has to be divided into
 but you can also specify the number of partitions when creating an RDD.
These partitions of an RDD is distributed across all the nodes in the network.

Creating an RDD
Creating an RDD is easy, it can be created either from an external file or
 by parallelizing collections in your driver. For example,

val rdd = sc.textFile("/some_file",3)
val lines = sc.parallelize(List("this is","an example"))
The first line creates an RDD from an external file,
and the second line creates an RDD from a list of Strings.

 Note that the argument ‘3’ in the method call sc.textFile() specifies the number of partitions
that has to be created. If you don’t want to specify the number of partitions,
 then you can simply call sc.textFile(“some_file”).

Actions/Transformations
There are two types of operations that you can perform on an RDD- Transformations and Actions.
Transformation applies some function on a RDD and creates a new RDD,
 it does not modify the RDD that you apply the function on.(Remember that RDDs are resilient/immutable).
 Also, the new RDD keeps a pointer to it’s parent RDD.


When you call a transformation, Spark does not execute it immediately, instead it creates a lineage.
 A lineage keeps track of what all transformations has to be applied on that RDD,
including from where it has to read the data. For example, consider the below example


val rdd = sc.textFile("spam.txt")
val filtered = rdd.filter(line => line.contains("money"))
filtered.count()
sc.textFile() and rdd.filter() do not get executed immediately,
it will only get executed once you call an Action on the RDD - here filtered.count().
An Action is used to either save result to some location or to display it.
You can also print the RDD lineage information
by using the command filtered.toDebugString(filtered is the RDD here).

RDDs can also be thought of as a set of instructions that has to be executed,
first instruction being the load instruction.
Caching
You can cache an RDD in memory by calling rdd.cache(). When you cache an RDD,
 it’s Partitions are loaded into memory of the nodes that hold it.


Caching can improve the performance of your application to a great extent.
In the previous section you saw that when an action is performed on a RDD,
it executes it’s entire lineage.
Now imagine you are going to perform an action multiple times on the same RDD
which has a long lineage, this will cause an increase in execution time.
Caching stores the computed result of the RDD in the memory thereby e
liminating the need to recompute it every time. You can think of caching as
if it is breaking the lineage, but
it does remember the lineage so that it can be recomputed in case of a node failure.

         
                         or

Resilient Distributed Datasets (RDDs)
RDDs are the main logical data unit in Spark. They are a distributed collection of objects,
 which are stored in memory or on disks of different machines of a cluster.
A single RDD can be divided into multiple logical partitions so that
these partitions can be stored and processed on different machines of a cluster.

RDDs are immutable (read-only) in nature. You cannot change an original RDD,
but you can create new RDDs by performing coarse-grain operations, like transformations, on an existing RDD.

-------------------------------------------------------------------------------------------------------
Q:-Why do we need RDD in Spark?
The key motivations behind the concept of RDD are-

Iterative algorithms.
Interactive data mining tools.
DSM (Distributed Shared Memory)

The main challenge in designing RDD is defining a program interface that provides fault tolerance efficiently.
To achieve fault tolerance efficiently,RDDs provide a restricted form of shared memory,


------------------------------------------------------------------------------------------------------
Q:- Features of RDD

Ans:-Partitioning
In-memory Computation
Immutability
Lazy Evaluation

Distributed
Resilient

-----------------------------------------------------------------------------------------------------

There are two basic operations which can be done on RDDs. They are:
Ans:-
Transformations
Actions

Transformations: These are functions which accept existing RDDs as the input and outputs one or more RDDs.
 The data in the existing RDDs does not change as it is immutable.
 Some of the transformation operations are shown in the table given below:

Functions Description
map()         Returns a new RDD by applying the function on each data element
filter() Returns a new RDD formed by selecting those elements of the source on which the function returns true
reduceByKey() Used to aggregate values of a key using a function
groupByKey() Used to convert a (key, value) pair to (key, <iterable value>) pair
union()         Returns a new RDD that contains all elements and arguments from the source RDD
intersection() Returns a new RDD that contains an intersection of elements in the datasets
These transformations are executed when they are invoked or called.
Every time transformations are applied, a new RDD is created.

Actions: Actions in Spark are functions which return the end result of RDD computations.
 It uses a lineage graph to load the data onto the RDD in a particular order.
 After all transformations are done, actions return the final result to the Spark Driver.
Actions are operations which provide non-RDD values. Some of the common actions used in Spark are:

Functions Description
count()         Gets the number of data elements in an RDD
collect() Gets all data elements in the RDD as an array
reduce() Aggregates data elements into the RDD by taking two arguments and returning one
take(n)         Used to fetch the first n elements of the RDD
foreach(operation) Used to execute operation for each data element in the RDD
first() Retrieves the first data element of the RDD

---------------------------------------------------------------------------------------------------
Q:-How many way to creat RDD?
Ans:-Creating an RDD
An RDD can be created in three ways:

1-By loading an external dataset
You can load an external file into an RDD.
The types of files you can load are csv, txt, JSON, etc.
 Here is an example of loading a text file into an RDD.

2-By parallelizing the collection of objects
When Spark’s parallelize method is applied on a group of elements, a new distributed dataset is created.
This is called an RDD.

3-By performing transformations on existing RDDs
One or more RDDs can be created by performing transformations on the existing RDDs.
The below figure shows how a map() function can be used.


----------------------------------------------------------------------------------------------------
 Q:-What is Map transformation operation in Apache Spark?
What is the need for the Map transformation?
What processing can be done in the Map in Spark explain with example?

Ans:-Map is a transformation applied to each element in a RDD and it provides a new RDD as a result.
In Map transformation, user-defined business logic will be applied to all the elements in the RDD.
It is similar to FlatMap, but unlike FlatMap Which can produce 0, 1 or many outputs,
Map can only produce one to one output.
Map operation will transforms an RDD of length N into another RDD of length N.

A——->a
B——->b
C——->c
Map Operation

Map transformation will not shuffle data from one partition to many. It will keep the operation narrow.

Q:-Explain the flatMap() transformation in Apache Spark.?
ans:-









---------------------------------------------------------------------------------------------

Q:- Diffrence between dataFrame and dataSet?
ans:-











-----------------------------------------------------------------------------------------------
Q:-What is diffrence map and FlatMap?
ans:-











------------------------------------------------------------------------------------------------------
Q:- What is diffrence between groupBy  key and reduce By key?
Ans:-











--------------------------------------------------------------------------------------------------------
Q:-How RDD fault Tolerence?
Ans:-










-----------------------------------------------------------------------------------------------------------
Q:- Where we need to map Partition?
Ans:-










--------------------------------------------------------------------------------------------------------
Q:-How to reduce no of partitions?
Ans:-








-------------------------------------------------------------------------------------------------------
Q:-How we will read file formate?
Ans:-







--------------------------------------------------------------------------------------------------------
Q:-What are the type of join in spark?
Ans:-









-------------------------------------------------------------------------------------------------------
Q:-If hive job is going to taking much time than what are the step you are going to take?
Ans:-








-----------------------------------------------------------------------------------------------------
Q:-How to cptimize the query?
Ans:-







------------------------------------------------------------------------------------------------------
Q:-How we will do unit testing ?

Ans:-





-----------------------------------------------------------------------------------------------------
Q:-If any partitioned is corrupt than how to handle ?
Ans:-








-----------------------------------------------------------------------------------------------------
Q:- Diffrence between DAG and Lenage?
Ans:-










-----------------------------------------------------------------------------------------------------
Q:-How to increased partitioned?
Ans:-









---------------------------------------------------------------------------------------------------
Q:-If i am applying aggregation after grouping what rdd we have to use?
Ans:-







----------------------------------------------------------------------------------------------------
Q:-Diffrence between order by and sort By?
Ans:-












------------------------------------------------------------------------------------------------------
======================================================================================================

    Hive Interview qustion
=======================================================================================================

Q:-What is Hive?








-----------------------------------------------------------------------------------------------------
Q:-Where to use hive?
Ans:-







-------------------------------------------------------------------------------------------------------
Q:- what are the feature of hive?
Ans:-







--------------------------------------------------------------------------------------------------------
Q:-What are tabular function in Hive?
Ans:-









--------------------------------------------------------------------------------------------------------
Q:-How to delete some data in hive?
Ans:-








--------------------------------------------------------------------------------------------------------
Q:-What are the table variable in HIve?
Ans:-









---------------------------------------------------------------------------------------------------------
Q:-Suppose if we drop internal table what will happen?
Ans:-








--------------------------------------------------------------------------------------------------------
Q:-Suppose if we drop external table than what will happen?








---------------------------------------------------------------------------------------------------------
Q:-What are the partitioned ? Where you implemented in your project?
Ans:-









--------------------------------------------------------------------------------------------------------
Q:-How many type of partitioned in Hive?
Ans:-









--------------------------------------------------------------------------------------------------------
Q:-What type of partitioned we should go?
Ans:-








-------------------------------------------------------------------------------------------------------
Q:-Why we use bucketing?
Ans:-









-------------------------------------------------------------------------------------------------------
Q:-What are file formate in Hive?
Ans:-











======================================================================================================




Tuesday, 20 March 2018

What is log4j

What is log4j ? How can we implement it? Explain log4j property ?

 

1. Log4j is logging framework developed by Apache
2. There are few steps to implement it into the project .

Step1 : download log4j.jar from apache web site and set into class path . 


Step 2 : Create one log4j.properties and set into class path. 

log4j prop file should have following value with naming convention as shown below  :

log4j.rootLogger=ERROR, stdout, logfile // stdout to pring on console  , logfile to print log into file

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n

log4j.appender.logfile=org.apache.log4j.RollingFileAppender
log4j.appender.logfile.File=logs/sample.log
log4j.appender.logfile.MaxFileSize=5MB
log4j.appender.logfile.MaxBackupIndex=3
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n

log4j.logger.com.sample=DEBUG


That's it .. Here I have used two type of  appender  , console and file but there lot more also available to configure. Appenders are which helps to print log.

Converiosn patterns Layout decide the pattern of printing . d - date , p - type , c - class name , m - message , n - name

And last line is used for filtering and type according to your packages to make the decision of log printing.
Here under com.sample package all classes debug log message will print if log4j Logger has set into class as following :

private static Logger _log= Logger.getLogger("com.sample.tutorials.main.SpringMain");
_log.debug("test debug");

Note : XML (log4j.xml) is also another way to configure the log4j logger. 

Empty and null check in JSTL JAVA

Difference between empty and null check of JSTL Core tag

< c:if test="${param.name != null}">< /c:if>


The above tag is to check whether parameter name is null or not. If it is not null then it prints out the sentence and it prints nothing if the value is null.

< c:if test="${param.name ne null}">< /c:if>

A test shows that parameter name contains value of ${param.name} via "ne" operator

Above codes are working similarly with the previous codes. We just want to test JSP notation “ne” for not equal function as what “!=” does.

< c:if test="${not empty(param.name)}">< /c:if>



A test shows that parameter name contains value of ${param.name} via empty operator.

Above codes are another feature of JSP notation to check a value whether it is empty or not. It is basically pretty much the same as the second and third tag do. The difference lies on the value null and empty. 
  
So What is the difference between empty and null? 
  
Well, empty is not necessarily be null but null always be empty. Null means that the variable is simply not existed while empty means that the variable is existed and initialized but it contains nothing. Please be careful when dealing with null values as it may cause you the famous NullPointerException.

Friday, 16 March 2018

How to Debug INSE Eclipse server or OutSide Eclpise Setting

Add following user variable
                CATALINA_BASE: D: \Liferay Developer Studio 1.5\liferay-portal-6.2.10-ee-ga1\tomcat-7.0.25
                CATALINA_HOME: D: \Liferay Developer Studio 1.5\liferay-portal-6.2.10-ee-ga1\tomcat-7.0.25
                JAVA_HOME:    C:\jdk1.7.0_20\bin
3. Go to following path:
D:\Liferay Developer Studio 1.5\liferay-portal-6.1.10-ee-ga1\tomcat-7.0.25\bin

4. Open startup.bat file
5. Add following line on top of this file
set JPDA_ADDRESS=8000
set JPDA_TRANSPORT=dt_socket

6. Chance executable command as below on last of this file(user jpda start to start sever into jpda debug mode on port 8000)
call "%EXECUTABLE%" jpda start %CMD_LINE_ARGS%

Steps to make configuration into eclipse:
1. Open Eclipse
2. Go to Run Menu -> Debug configuration
3. Go to Remote java application on opend box at left side
4. Add project as “your  portlet” (select from browse button in connect tab)
5. Select connect type as Standard (socket attach)
6. Set connection property: localhost
7. Set Port:8000
8. Clickon Common Tab
9. Select Debug check box

Now click on Debug  to debug remote server on port 8000





Friday, 29 December 2017

Git (version control)

Q-What is Git ??
Ans- Git is a Distributed Version Control tool that supports distributed non-linear workflows by providing data assurance for developing quality software.

The motive of Git is to manage a project or a set of files as they change over time. Git stores this information in a data structure called a Git repository. The repository is the core of Git.
To be very clear, a Git repository is the directory where all of your project files and the related metadata resides.

Generally We have two type of version control system
1-Centralized Version Control System (CVCS(SVN-Subversion)).
2-Distributed Version Control System (DVCS(Git)).

1-Centralized VCS(SubVersion)

Centralized version control system (CVCS) uses a central server to store all files and enables team collaboration. It works on a single repository to which users can directly access a central server.

Note- Here repository is new word---Repository  is place where you will have all your code.


The repository in the above diagram indicates a central server that could be local or remote which is directly connected to each of the programmer’s workstation.
Every programmer can extract or update their workstations with the data present in the repository or can make changes to the data or commit in the repository. Every operation is performed directly on the repository.

it has some major drawbacks. Some of them are:
  • It is not locally available; meaning you always need to be connected to a network to perform any action.
  • Since everything is centralized, in any case of the central server getting crashed or corrupted will result in losing the entire data of the project.


Distributed VCS
These systems do not necessarily rely on a central server to store all the versions of a project file.
In Distributed VCS, every contributor has a local copy or “clone” of the main repository i.e. everyone maintains a local repository of their own which contains all the files and metadata present in the main repository.

You will understand it better by referring to the diagram below:
Advantage suppose your server fails its fine you have local copy
so every machine every developer have own copy so this called distrributed
In this digram we are distrubting repository
As you can see in the above diagram, every programmer maintains a local repository on its own, which is actually the copy or clone of the central repository on their hard drive. They can commit and update their local repository without any interference.
They can update their local repositories with new data from the central server by an operation called “pull” and affect changes to the main repository by an operation called “push” from their local repository.
The act of cloning an entire repository into your workstation to get a local repository gives you the following advantages:
  • All operations (except push & pull) are very fast because the tool only needs to access the hard drive, not a remote server. Hence, you do not always need an internet connection.
  • Committing new change-sets can be done locally without manipulating the data on the main repository. Once you have a group of change-sets ready, you can push them all at once.
  • Since every contributor has a full copy of the project repository, they can share changes with one another if they want to get some feedback before affecting changes in the main repository.
  • If the central server gets crashed at any point of time, the lost data can be easily recovered from any one of the contributor’s local repositories.
After knowing Distributed VCS, its time we take a dive into what is Git.

Better how? - Better to manage your code.
How to manage? - Helps maintaining and keeping track of different versions of your code.
Versions?- When you code, you tend to keep adding let’s say modules, maybe bug fixes etc. basically meaning changing your initial code. But sometimes, new addition do not work and you want to go back to the state where it worked.
In such cases, tools like Git comes in handy, because it had been maintaining your changes inn code and preserving every version. So, you can revert back anytime.
How does Git identify changes?- So, every time you make a change in your code, you need to “commit”. Each commit gives you a snapshot of your code.
Where does Git store all these changes?- In a repository. Repository can be thought of a data space where you can store all your files related to your project.
Mostly, you work with two repositories - local, which resides in your system and remote/central which mostly lies on a remote machine - something like GitHub.
Ah! Here comes GitHub!
Technical definition - It is a website where you can host your repository and share them with people.
Friendly definition - It is a Facebook for programmers. Instead of pictures and 
Differences between GIt vs SVN:-
The key difference is that it is decentralized. Imagine you are a developer on the road, you develop on your laptop and you want to have source control so that you can go back 3 hours.
With Subversion, you have a Problem: The SVN Repository may be in a location you can't reach (in your company, and you don't have internet at the moment), you cannot commit. If you want to make a copy of your code, you have to literally copy/paste it.