Colonialism was one factor. It is not clear how India would have imbibed and benefited from the Enlightenment and Industrial Revolution in the absence of British presence in the subcontinent. Just a reminder that British presence started in the very late 18th century, but their true hegemony was for 90 years, between 1857 and 1947. Japan had American presence, but no hegemony and is a good counterpoint to those who claim that colonialism has benefits. A few things to note: without the British, there may never have been a coherent whole of "India" as we know today. Counterfactuals are hard, but it is hard to assume that there would necessarily have been an Indian Bismarck or Garibaldi had there been no British hegemony. Different regions and kingdoms in India would have developed differently. You would have had a few Japans, notably in the South, West coast, Northeast, and parts of the Gangetic heartland: regions which were always wealthy and/or politically stable.
Whether the economy would have declined less in the absence of colonialism is really impossible to answer. But to cite colonialism as the only reason is wrong, especially since the decline coincides roughly with the scientific and industrial revolutions that occurred in Western Europe and later, America.