Matcha is traditionally made with water. Using milk, like in a matcha latte, is a newer trend. Look up a traditional match recipe using a whisk. Note, I know that most people do this with ceremonial grade matcha, but I'm not sure if people do this with lower grade matcha (for example, the bag you might buy at Costco). It sounds like you are probably not using ceremonial grade, so I can't vouch for how good it will taste. However it's still worth experimenting with.

"Cold brew" matcha doesn't really make sense. The goal of cold brew is to extract flavor from coffee/tea leaves over a long period of time using room temp/cold water. However matcha isn't really meant to be extracted, it's meant to be suspended in a liquid and drunk. If you want a "cold matcha drink" rather than specifically cold brew, I would try just throwing some matcha with some cold water in a mason jar, pop the lid on, and shake it up. Then just drink immediately.

but I want to drink more water

Do you want to drink more water, or less milk?

If your milk consumption doesn't bother you, then milk is mostly water. It all counts. Have your matcha however you like your matcha.

I want to start drinking tea more instead of coffee

Not sure if you are aware of it, but matcha has pretty high caffeine content. It's less than in coffee, but more than in other types of tea. One gram of matcha powder contains approximately 20-45mg of caffeine. So, if you prepare a cup of matcha using 2 grams of matcha powder, you could get up to 90 mg of caffeine. Tea leaves being naturally grown things, it's impossible to know how much caffeine is in each tea leaf.

In case you want to reduce your caffeine intake even more, you might want to try other Japanese green teas. As you like matcha, your taste buds might agree with other Japanese green teas as well.

Also I have a cold brew coffee maker, could I put the powder in the filter as I submerge it? How long would it last in the fridge (assuming I can)?

One tea brewing method you might want to look into is grandpa style tea brewing. To grandpa style brew tea, you simply put some tea leaves into some kind of container, such as mug or water bottle, and add some water. As you drink the tea throughout the day, you just add more water over the leaves as needed, basically keeping the leaves submerged in water the whole day. Some teas are better suited for grandpa style brewing than others. Japanese tea leaves are usually broken leaves, and broken leaves quickly turns the tea liquor bitter/astringent and as such might not be suited for grandpa style brewing.

There are couple tea communities that might be helpful in your tea journey:

• !tea@possumpat.io
• !tea@lemmy.zip

You probably have been drinking matcha lattes, which contains milk. Normal matcha is just water.

https://www.wikihow.com/Make-Matcha-Tea

I had never heard of matcha tea with milk before, only matcha with water. I guess making a latte with matcha tea would use milk in it, but you are absolutely fine just using hot water, as that is the traditional method.

Cold brewing, though, I have no idea for.

I always use Boiled water when using ceremonial grade matcha. There is matcha sets you can get that comes with a special bowl as well as a matcha scoop and a Chasen which is a special whisk just for matcha.

My wife makes macha tea with just water. It's not very tasty.

I prefer it straight with water, myself. Sometimes it’s hard to find places that will sell a straight matcha water. It’s important to use at least culinary grade if not ceremonial, and order your matcha from a reputable distributor.

I like Tenzo tea, but it’s overpriced. Enso or something like that is what’s in my cabinet these days.