Any substantial numbers of Japanese first came to what's now the US only starting in 1865. That's when Hawai'i saw a large number of migrant works, arriving to the islands. This date marks the beginning of a period of a profitable sugarcane market from Hawai'i to the States after the US Civil War caused a disruption in the market on the mainland.
But what you're seeing more of initially is not immigrants so much as migrant workers operating under labour contracts. Also of course at this point the Kingdom of Hawai'i is not part of the United States. Still, it's worth brining up since it was the start of what became America's earliest Japanese communities.
Japanese immigration didn't really pick up until a bit later, with the first real wave of Japanese immigration happening in 1885, again to Hawai'i. Even later on, well into the 20th century, a large portion of Japanese communities were to be found in Hawai'i , with only a fraction in places like California. They definitely existed on the mainland, but the number have always been small compared to Japanese Hawai'ians.
That said, there are a few cases of very early immigration (in the sense of becoming US citizens). One example is Hamada Hikozō (浜田彦蔵 ) who is cited as being the first, naturalising in 1858. His story isn't a typical one for the time though. He was shipwrecked on the West Coast in the early 50's and over the following years had stayed in the US (for the most part).
/u/keyilan has excellent observations. I can add that on the Comstock - the famed Nevada mining district that included Virginia City with a peak population in the 1870s of about 25k - Chinese immigrants were ubiquitous but Japanese immigrants were nearly non existent. In the manuscript census documents of 1860, 70 and 80 and in related documents from the 1870s, Chinese immigrants number in the hundreds and probably peaked at nearly 2k, but among all the people who arrived there from Asia, only two - cooks who appear in 1880 - were from Japan.
That said, an excavation of Virginia City's Chinatown revealed some Japanese artifacts. It appears that importers offering material to fellow Chinese immigrants but also exotic products to other Comstock residents drew from Japanese sources as well as those from China.