The Communist Party traces its lineage back to 1925, when Ho Chi Minh established the Vietnamese Revolutionary Youth League, commonly referred to as Thanh Niên. Thanh Niên sought to make use of patriotism in an effort to end the colonial occupation of the country by France. The group sought political and social objectives—national independence and the redistribution of land to working peasants. Thanh Niên was designed to prepare the ground for a revolutionary armed struggle against the French occupation.

In 1928 the headquarters of the Thanh Niên organization in Canton was forced underground by Chinese nationalists led by Kuomintang (KMT). This led to a national breakdown within the organization,  which lead indirectly to a split within Thanh Niên. On 17 June 1929, more than 20 delegates from cells throughout the Tonkin region held a conference in Hanoi, where they declared the dissolution of Thanh Nien and the establishment of a new organization called the Communist Party of Indochina (ICP). The other faction of Thanh Niên, based in the central and southern administrative districts of the country, renamed themselves the Communist Party of Annam in late 1929. The two organizations spent the rest of 1929 engaged in polemics against one another in an attempt to gain a position of hegemony over the radical Vietnamese liberation movement. A third Vietnamese Communist Party emerged around this time, called the League of Indochinese Communists (Vietnamese: Đông Dương Cộng sản Liên Đoàn), which was not connected with Thanh Niên. The League of Indochinese Communists had its roots in another national liberation group which had existed in parallel with Thanh Niên, and saw itself as a rival to the latter.

The two warring splinters of Thanh Niên joined with individual members of a third Marxist group called the Communist Party of Vietnam, founded by Phan Bội Châu at a "Unification Conference" held in Hong Kong from 3–7 February 1930. At a later conference, the party was renamed the Indochinese Communist Party (Vietnamese: Đông Dương Cộng sản Đảng, often abbreviated to ICP). During its first five years of existence, the ICP attained a membership of about 1500 and had large contingent of sympathizers. Despite the group's small size, it exerted an influence in a turbulent Vietnamese social climate. Poor harvests in 1929 and 1930 and an onerous burden of debt served to radicalize many peasants. In the industrial city of Vinh, May Day demonstrations were organized by ICP activists, which gained critical mass when the families of the semi-peasant workers joined the demonstrations to express their dissatisfaction with the economic circumstances they faced.

As three May Day marches grew into mass rallies, French colonial authorities moved in to quash what they perceived to be dangerous peasant revolts. Government forces fired upon the assembled crowds, killing dozens of participants and inflaming the population. In response, local councils were organized in villages in an effort to govern themselves locally. Repression by the colonial authorities began in the fall 1931, and about 1300 people were eventually killed by the French and many more were imprisoned or deported as government authority was reasserted and the ICP was effectively wiped out in the region. General Secretary Tran Phu and many leaders of the Central Committee were arrested and killed. Lê Hồng Phong was assigned to Communist International to restore the movement. The party was restored in 1935, and Lê Hồng Phong was elected its General Secretary. In 1936, Hà Huy Tập was appointed General Secretary instead of Lê Hồng Phong, who returned to the country to restore the Central Committee.

The Second World War drastically weakened the grip of France on Indochina. The fall of France to Nazi Germany in June 1940 and the subsequent collaboration of Vichy France with the Axis powers of Germany and Japan served to delegitimize French claims of sovereignty. The European war made colonial governance from France impossible and Indochina was occupied by Japanese forces.

At the beginning of the war, the Indochinese Communist Party instructed its members to go into hiding in the countryside as an underground organization. Despite this, more than 2,000 members of the party, including many key leaders, were rounded up and arrested. Party activists were particularly hard hit in the southern region of Cochinchina, where the previously strong organization was wiped out by arrests and killings. After an uprising in Cochinchina in 1940, most of the Central Committee leaders, including Nguyen Van Cu (General Secretary) and Hà Huy Tập, were arrested and killed, and Lê Hồng Phong was deported to Côn Đảo and died later. A new party leadership, which included Trường Chinh, Phạm Văn Đồng, and Võ Nguyên Giáp emerged. Together with Ho Chi Minh, these would provide a unified leadership over the next four decades,

Party leader Ho Chi Minh returned to Vietnam in February 1941 and established a military organization known as the League for the Independence of Vietnam (Vietnamese: Việt Nam Độc Lập Đồng Minh Hội, commonly referred to as "Vietminh"). It was the most uncompromising fighting force against the Japanese occupation, and gained popular recognition and legitimacy in an environment that would develop into a political vacuum. Despite its position as the core of the Vietminh organization, the Indochinese Communist Party remained very small through the war years, with an estimated membership of between 2,000 and 3,000 in 1944.