Wilde was put into prison at a particularly grim time in the history of the English prison system, between the Prison Act of 1877 and the reforms of the 1920s. Essentially, the goal of the Prison Act and its concomitant rules and regulations was to change the approach for convicted criminals from punishments (especially corporal punishments like whippings) to incarceration and strict monitoring of conduct in hopes of driving out their baser instincts. The British prison system was centralized during this time under a single Prison Commission, and strict rules were put into place about prisoner separation, silence, hard labor, difficult sleeping and living conditions, and general austerity as a national approach.

The result of this was that when Wilde came to prison (first Pentonville Prison), he was met with a system that was based upon that of the United States' penitentiary model -- strict separation of prisoners, silence among them, in order to prevent "corruption" by fellow prisoners. Wilde was made to walk endlessly on a treadmill for hours and to separate old rope into small strands called oakum that are used for plugging holes in ships' hulls, and was not allowed to have any reading material other than the Bible and The Pilgrim's Progress. His condition was very poor, as Wilde was a man of delicate health, and he suffered greatly.

He was soon moved to Wandsworth Prison, where conditions were extremely bad. In a well-known incident that may go to your question, another prisoner whispered to Wilde "in the hoarse prison-voice men get from long and compulsory silence" during the exercise period in the prison yard: "I am sorry for you: it is harder for the likes of you than it is for the likes of us" This moved Wilde, who had been badly treated by all he'd encountered up to that point. Wilde responded: "No, my friend, we all suffer alike." and was caught by a guard. Because he wished to spare his fellow prisoner any punishment, Wilde told the prison warden that he had begun the conversation, and was confined to a dark, solitary cell for three days with nothing but bread and water. Wilde later collapsed in chapel, and ruptured his ear drum as a result of the fall, then spent two months in the infirmary recovering.

Liberal MP Richard Haldane found out about Wilde's condition and had him transferred to Reading Gaol, where things were slightly better -- Wilde was provided with writing materials and some of his requested reading materials, but only due to the efforts and continued pressure of Haldane. Wilde wrote to the Daily Chronicle:

The present prison system seems almost to have for its aim the wrecking and the destruction of the mental faculties.... Deprived of books, of all human intercourse, isolated from every humane and humanising influence, condemned to eternal silence, robbed of all intercourse with the external world, treated like an unintelligent animal, brutalised below the level of any of the brute-creation, the wretched man who is confined in an English prison can hardly escape becoming insane.

However, at Reading Gaol, Wilde was assigned to work in the garden and act as the schoolmaster's orderly -- freeing him somewhat from the kinds of hard labor that he had been doing earlier, and that other prisoners would have had to do. He did still have to take turns at hard labor, though -- turning a crank, and again, picking oakum, but less often than the others. This, nevertheless, negatively impacted his health greatly. According to a news report at the time, Reading Gaol's governor (warden) Henry Isaacson, boasted that he "knocked the nonsense out of" Wilde, and Wilde himself wrote to Andre Gide that he found Isaacson "very harsh because he lacked imagination."

In 1896, a new governor, Major J.O. Nelson, succeeded Isaacson, and Wilde said that "this new personality that has altered every man's life in this place"; later, Wilde called him "the most Christ-like man I ever met." Nelson relaxed the restrictions, giving Wilde the ability to write letters (with the proviso that the governor be able to read them at the end of every day), and less hard labor time. But, by now, the damage had been done.

So, to answer your question: it is difficult to say whether Wilde's prison time objectively was harder than other people's. He certainly was unlucky in his timing of going to prison and, unlike many prisoners around him, he was not at all inured to the difficulties of prison life. He definitely suffered, there can be no doubt, but it is hard to compare levels of suffering, in the end.

Sources:

Bailey, Victor, English Prisons, Penal Culture, and the Abatement of Imprisonment 1895-1922

Ellman, Richard, Oscar Wilde 1985