Python – How to join two pandas DataFrames with similar indexes of different sizes

How to join two pandas DataFrames with similar indexes of different sizes… here is a solution to the problem.

How to join two pandas DataFrames with similar indexes of different sizes

I want

to merge two dataframes where the individual indexes exist in a sorted manner but appear differently in the dataframe I want to merge.

frame1 = pd. DataFrame([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], index=['A','B','B','C','C','C','D','E','E','F'])
frame2 = pd. DataFrame([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], index=['A', 'A', 'B', 'C', 'C', 'D', 'D', 'E', 'F', 'F'])
frame1.columns =['Hi']
frame2.columns =['Bye']

frame1
Out[160]: 
   Hi
A   1
B   2
B   3
C   4
C   5
C   6
D   7
E   8
E   9
F  10

frame2
Out[161]: 
   Bye
A    1
A    2
B    3
C    4
C    5
D    6
D    7
E    8
F    9
F   10

Expected output:

    Bye    Hi
A   1.0   1.0
A   2.0   NaN
B   3.0   2.0
B   NaN   3.0
C   4.0   4.0
C   5.0   5.0
C   NaN   6.0
D   6.0   7.0
D   7.0   NaN
E   8.0   8.0
E   NaN   9.0
F   9.0  10.0
F  10.0   NaN

Can’t seem to find any correct combination of concat or join to do this. Is there any way?

Solution

Okay, let’s build a new key here using comcount

s1=frame1.set_index(frame1.groupby(level=0).cumcount(),append=True)  
s2=frame2.set_index(frame2.groupby(level=0).cumcount(),append=True)

pd.concat([s2,s1],1).reset_index(level=1,drop=True)
Out[364]: 
    Bye    Hi
A   1.0   1.0
A   2.0   NaN
B   3.0   2.0
B   NaN   3.0
C   4.0   4.0
C   5.0   5.0
C   NaN   6.0
D   6.0   7.0
D   7.0   NaN
E   8.0   8.0
E   NaN   9.0
F   9.0  10.0
F  10.0   NaN

From piR (Great Solution with Custom Features).

def add_cumcount_level(df):
    return df.set_index(df.groupby(level=0).cumcount(), append=True)

pd.concat(map(add_cumcount_level, [frame1, frame2]), axis=1)

Related Problems and Solutions