["python","pandas","dataframe","concatenation","nan"]
pandas concat generates nan values

I am curious why a simple concatenation of two dataframes in pandas: initId.shape # (66441, 1) initId.isnull().sum() # 0 ypred.shape # (66441, 1) ypred.isnull().sum() # 0 of the same shape and both without NaN values foo = pd.concat([initId, ypred], join='outer', axis=1) foo.shape # (83384, 2) foo.isnull().sum() # 16943 can result in a lot of NaN values if joined. How can I fix this problem and prevent NaN values being introduced? Trying to reproduce it like aaa = pd.DataFrame([0,1,0,1,0,0], columns=['prediction']) bbb = pd.DataFrame([0,0,1,0,1,1], columns=['groundTruth']) pd.concat([aaa, bbb], axis=1) failed e.g. worked just fine as no NaN values were introduced.
I think there is problem with different index values, so where concat cannot align get NaN: aaa = pd.DataFrame([0,1,0,1,0,0], columns=['prediction'], index=[4,5,8,7,10,12]) print(aaa) prediction 4 0 5 1 8 0 7 1 10 0 12 0 bbb = pd.DataFrame([0,0,1,0,1,1], columns=['groundTruth']) print(bbb) groundTruth 0 0 1 0 2 1 3 0 4 1 5 1 print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 0 NaN 0.0 1 NaN 0.0 2 NaN 1.0 3 NaN 0.0 4 0.0 1.0 5 1.0 1.0 7 1.0 NaN 8 0.0 NaN 10 0.0 NaN 12 0.0 NaN Solution is reset_index if indexes values are not necessary: aaa.reset_index(drop=True, inplace=True) bbb.reset_index(drop=True, inplace=True) print(aaa) prediction 0 0 1 1 2 0 3 1 4 0 5 0 print(bbb) groundTruth 0 0 1 0 2 1 3 0 4 1 5 1 print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 0 0 0 1 1 0 2 0 1 3 1 0 4 0 1 5 0 1 EDIT: If need same index like aaa and length of DataFrames is same use: bbb.index = aaa.index print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 4 0 0 5 1 0 8 0 1 7 1 0 10 0 1 12 0 1